测量编程语言分布的影响
Measuring The Impact Of Programming Language Distribution
AI总结：
针对现有代码评估基准语言覆盖不全的问题，提出BabelCode框架与TP3数据集，并验证平衡训练数据中的语言分布能显著提升模型在低资源语言上的$pass@k$表现。
摘要：
当前用于评估神经代码模型的基准仅关注一小部分编程语言，排除了许多流行的语言，如Go或Rust。为改善这一问题，我们提出了BabelCode框架，用于在任何语言中对任何基准进行基于执行的评估。BabelCode能够对模型在内存、运行时和单个测试用例结果方面的定性性能展开新的调查。此外，我们还提出了一个新的代码翻译数据集，称为Translating Python Programming Puzzles (TP3)，该数据集源自Python Programming Puzzles (Schuster et al. 2021)基准，涉及将专家级Python函数翻译为任何语言。借助BabelCode和TP3基准，我们研究了在训练数据集中平衡14种语言的分布是否能提升大型语言模型在低资源语言上的性能。与基线相比，在平衡语料库上训练模型，使得所有任务和语言上的$pass@k$平均提高了12.34%。我们发现，该策略在低资源语言上实现了66.48%更好的$pass@k$，而代价仅为高资源语言下降12.94%。在我们的三项翻译任务中，该策略平均使低资源$pass@k$提高了30.77%，而高资源$pass@k$则下降了19.58%。
英文摘要：
Current benchmarks for evaluating neural code models focus on only a small subset of programming languages, excluding many popular languages such as Go or Rust. To ameliorate this issue, we present the BabelCode framework for execution-based evaluation of any benchmark in any language. BabelCode enables new investigations into the qualitative performance of models' memory, runtime, and individual test case results. Additionally, we present a new code translation dataset called Translating Python Programming Puzzles (TP3) from the Python Programming Puzzles (Schuster et al. 2021) benchmark that involves translating expert-level python functions to any language. With both BabelCode and the TP3 benchmark, we investigate if balancing the distributions of 14 languages in a training dataset improves a large language model's performance on low-resource languages. Training a model on a balanced corpus results in, on average, 12.34% higher $pass@k$ across all tasks and languages compared to the baseline. We find that this strategy achieves 66.48% better $pass@k$ on low-resource languages at the cost of only a 12.94% decrease to high-resource languages. In our three translation tasks, this strategy yields, on average, 30.77% better low-resource $pass@k$ while having 19.58% worse high-resource $pass@k$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
