CLARC：用于鲁棒代码搜索的C/C++基准测试
CLARC: C/C++ Benchmark for Robust Code Search
AI总结：
CLARC是一个用于测试鲁棒代码检索的C/C++基准测试，通过引入挑战性设置验证模型对词法特征的依赖。
摘要：
高效的代码检索对于开发者生产力至关重要，但现有的基准测试大多集中在Python上，很少对鲁棒性进行深入测试。为弥补这一差距，我们介绍了一个用于代码搜索数据集的自动化流程，并提出了CLARC，一个基于真实世界GitHub仓库的C/C++基准测试。CLARC包含1,245对查询代码用于评估和5,472对用于训练。该基准测试包含由LLM生成的自然语言查询，通过严格的评分和假设测试进行了验证。为了有效分析上下文要求，我们的流程首先确保代码可编译。然后根据依赖复杂性对代码片段进行分类，区分代码是否依赖自定义定义的类型或辅助函数。该流程还使CLARC能够通过引入具有挑战性的设置来测试检索鲁棒性，包括标识符匿名化和编译到低级语言如汇编和WebAssembly。在这些条件下，我们对六个最先进的模型的评估显示检索效果显著下降。实验结果突显了这些模型对词法特征的依赖而非代码语义理解。我们的数据集在https://huggingface.co/datasets/ClarcTeam/CLARC上公开可用。
英文摘要：
Efficient code retrieval is critical for developer productivity, yet existing benchmarks largely focus on Python and rarely stress-test robustness beyond superficial lexical cues. To address the gap, we introduce an automated pipeline for code search datasets and present CLARC, a C/C++ benchmark built from real-world GitHub repositories. CLARC contains 1,245 query-code pairs for evaluation and 5,472 pairs for training. The benchmark incorporates LLM-generated natural language queries validated through rigorous human scoring and hypothesis testing. To analyze contextual requirements effectively, our pipeline starts by ensuring code compilability. It then categorizes code snippets by dependency complexity, distinguishing whether the code relies on custom-defined types or helper functions. The pipeline also enables CLARC to stress-test retrieval robustness by introducing challenging settings, including identifier anonymization and compilation to low-level languages like Assembly and WebAssembly. Under these conditions, our evaluation of six state-of-the-art models reveals sharp drops in retrieval effectiveness. The experimental results highlight the models' persistent reliance on lexical features rather than code semantic understanding. Our dataset is publicly available at https://huggingface.co/datasets/ClarcTeam/CLARC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
