arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.07844cs.SEcs.CL
收藏

Mercury:面向代码大型语言模型的代码效率基准

Mercury: A Code Efficiency Benchmark for Code Large Language Models

Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);Sea AI Lab(Sea AI实验室)

Mingzhe Du, Anh Tuan Luu, Bin Ji, Qian Liu, See-Kiong Ng

AI总结:

本文提出首个代码效率基准 Mercury,包含 1,889 个 Python 任务及运行时百分位加权指标 Beyond,实验揭示 Code LLM 效率短板,并验证 DPO 相较 SFT 更适合作为效率优化基线。

摘要:

近期,在评估代码大型语言模型(Code LLM)方面取得了诸多进展,但现有基准主要关注生成代码的功能正确性,忽视了其计算效率的重要性。为填补这一空白,我们提出 Mercury,这是首个面向 Code LLM 的代码效率基准。它包含 1,889 个 Python 任务,每个任务都配有充足的解决方案,作为真实世界中的效率基线,从而能够全面分析运行时分布。基于该分布,我们引入了一个新指标 Beyond,通过计算运行时百分位加权的 Pass 分数,同时反映功能正确性和代码效率。在 Mercury 上,领先的 Code LLM 在 Pass 上可达到 65%,但在 Beyond 上低于 50%。鉴于理想情况下 Beyond 分数应与 Pass 分数保持一致,这表明尽管 Code LLM 在生成功能正确的代码方面表现出令人瞩目的能力,但其效率仍存在显著差距。最后,我们的实证实验表明,与监督微调(SFT)相比,直接偏好优化(DPO)是提升代码效率的稳健基线,为未来探索高效代码生成铺平了一条有前景的道路。我们的代码和数据可在 GitHub 获取:https://github.com/Elfsong/Mercury。

英文摘要:

Amidst the recent strides in evaluating Large Language Models for Code (Code LLMs), existing benchmarks have mainly focused on the functional correctness of generated code, neglecting the importance of their computational efficiency. To fill the gap, we present Mercury, the first code efficiency benchmark for Code LLMs. It comprises 1,889 Python tasks, each accompanied by adequate solutions that serve as real-world efficiency baselines, enabling a comprehensive analysis of the runtime distribution. Based on the distribution, we introduce a new metric Beyond, which computes a runtime-percentile-weighted Pass score to reflect functional correctness and code efficiency simultaneously. On Mercury, leading Code LLMs can achieve 65% on Pass, while less than 50% on Beyond. Given that an ideal Beyond score would be aligned with the Pass score, it indicates that while Code LLMs exhibit impressive capabilities in generating functionally correct code, there remains a notable gap in their efficiency. Finally, our empirical experiments reveal that Direct Preference Optimization (DPO) serves as a robust baseline for enhancing code efficiency compared with Supervised Fine Tuning (SFT), which paves a promising avenue for future exploration of efficient code generation. Our code and data are available on GitHub: https://github.com/Elfsong/Mercury.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑