CodeUltraFeedback：一个用于将大语言模型对齐到编码偏好的LLM-as-a-Judge数据集
CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences
AI总结：
本文提出CodeUltraFeedback数据集，利用LLM-as-a-Judge方法评估编码偏好对齐，并通过SFT和RLAIF（DPO）微调CodeLlama-7B-Instruct，使其对齐效果和功能正确性优于更大模型。
摘要：
评估大语言模型（LLMs）与用户定义的编码偏好的对齐情况是一项具有挑战性的工作，需要对LLMs的输出进行深入评估。现有方法和基准主要依赖自动化指标和静态分析工具，这些方法往往无法捕捉用户指令和LLM输出中的细微差别。为弥补这一不足，我们提出使用LLM-as-a-Judge方法来评估LLMs与编码偏好的对齐情况。基于该方法，我们提出了CodeUltraFeedback，这是一个旨在促进LLM对齐评估和改进的综合数据集。CodeUltraFeedback包含10,000条编码指令，每条指令都标注了来自14个不同LLM池生成的四个响应。这些响应由GPT-3.5作为评判者，根据五种不同的编码偏好进行排序，同时提供数值评分和详细的文本反馈。我们对CodeUltraFeedback的分析表明，GPT-3.5和GPT-4生成的响应通常比开放权重LLMs生成的响应更受偏好，这凸显了闭源模型与开放权重模型之间在对齐方面的显著差异。接着，我们探索了将CodeUltraFeedback用作反馈数据，通过监督微调（SFT）和基于AI反馈的强化学习（RLAIF）结合直接偏好优化（DPO）来微调和对齐CodeLlama-7B-Instruct。最终对齐后的CodeLlama-7B-Instruct模型在编码偏好对齐方面优于更大的LLMs，并且与原始instruct模型相比，在HumanEval+基准上展现出更好的功能正确性。因此，我们的贡献弥补了代码领域LLMs偏好调优的不足，并为自动化软件工程中模型对齐和RLAIF的进一步发展奠定了基础。
英文摘要：
Evaluating the alignment of large language models (LLMs) with user-defined coding preferences is a challenging endeavour that requires a deep assessment of LLMs' outputs. Existing methods and benchmarks rely primarily on automated metrics and static analysis tools, which often fail to capture the nuances of user instructions and LLM outputs. To address this gap, we propose using the LLM-as-a-Judge methodology to evaluate the alignment of LLMs with coding preferences. Based on this approach, we present CodeUltraFeedback, a comprehensive dataset designed to facilitate the evaluation and improvement of LLM alignment. CodeUltraFeedback consists of 10,000 coding instructions, each annotated with four responses generated from a diverse pool of 14 LLMs. These responses are ranked based on five distinct coding preferences using GPT-3.5 as a judge, providing both numerical scores and detailed textual feedback. Our analysis of CodeUltraFeedback reveals that responses from GPT-3.5 and GPT-4 are generally preferred over those from open-weight LLMs, highlighting significant differences in alignment between closed and open-weight models. In turn, we explore the usage of CodeUltraFeedback as feedback data to fine-tune and align CodeLlama-7B-Instruct using supervised fine-tuning (SFT) and reinforcement learning from AI feedback (RLAIF) with direct preference optimization (DPO). The resulting aligned CodeLlama-7B-Instruct model outperforms larger LLMs in terms of alignment with coding preferences and shows improved functional correctness on the HumanEval+ benchmark compared to the original instruct model. Therefore, our contributions bridge the gap in preference tuning of LLMs for code and set the stage for further advancements in model alignment and RLAIF in automated software engineering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
