BigCodeArena: 通过执行揭示更多可靠的代码生成人类偏好
BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
AI总结：
BigCodeArena通过执行揭示更多可靠的代码生成人类偏好，提出自动评分基准评估LLM编码质量。
摘要：
众包模型评估平台，如Chatbot Arena，能够从人类角度进行实时评估，以评估模型响应的质量。在编码领域，手动检查LLM生成内容的质量极具挑战性，因为这需要理解大量的原始代码并有意模拟代码执行。为此，我们引入了BigCodeArena，这是一个基于综合且实时执行环境的开源人类评估平台，用于代码生成。基于Chatbot Arena构建，BigCodeArena能够执行LLM生成的代码，并允许人类与执行过程和结果互动。我们收集了超过14,000个以原始代码为中心的对话会话，涵盖10个广泛使用的LLM，跨越10种语言和8种执行环境。在这些对话中，我们识别出超过4,700个多轮样本，具有成对的人类偏好。进一步分析揭示了在细粒度领域中LLM的未被探索的偏好，这些领域由任务、语言和框架特征化。为了系统地检查前沿LLM的代码理解和生成能力，我们基于收集的数据整理了两个基准，即BigCodeReward和AutoCodeArena。对于BigCodeReward，我们处理了4,700次对话，并评估了奖励模型与人类偏好的一致性。评估显示，大多数LLM在执行结果可用时在判断编码偏好方面表现更优。受这些发现的启发，我们提出了AutoCodeArena，一个自动Elo评分基准，旨在评估LLM的编码质量，而无需人类参与。我们发现，像GPT-5、Claude-Sonnet-4和Claude-Opus-4这样的专有LLM在最近新兴模型中仍然在代码生成性能方面领先。
英文摘要：
Crowdsourced model evaluation platforms, such as Chatbot Arena, enable real-time evaluation from human perspectives to assess the quality of model responses. In the coding domain, manually examining the quality of LLM-generated content is extremely challenging, as it requires understanding long chunks of raw code and deliberately simulating code execution. To this end, we introduce BigCodeArena, an open human evaluation platform for code generation backed by a comprehensive and on-the-fly execution environment. Built on top of Chatbot Arena, BigCodeArena enables the execution of LLM-generated code and allows humans to interact with the execution process and outcomes. We collected over 14,000 raw code-centric conversation sessions across 10 widely used LLMs, spanning 10 languages and 8 types of execution environments. Among these conversations, we identified more than 4,700 multi-turn samples with pairwise human preferences. Further analysis uncovers underexplored preferences of LLMs in fine-grained domains characterized by tasks, languages, and frameworks. To systematically examine code understanding and generation capabilities of frontier LLMs, we curated two benchmarks based on the collected data, namely BigCodeReward and AutoCodeArena. For BigCodeReward, we post-processed the 4,700 conversations and evaluated the consistency between reward models and human preferences. The evaluation shows that most LLMs have superior performance in judging coding preferences when the execution results are available. Inspired by these findings, we propose AutoCodeArena, an automatic Elo rating benchmark designed to assess the coding quality of LLMs without human involvement. We find that proprietary LLMs like GPT-5, Claude-Sonnet-4, and Claude-Opus-4 still lead in code generation performance among recent emerging models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
