在轻量级大语言模型中应用 RLAIF 进行含 API 调用的代码生成
Applying RLAIF for Code Generation with API-usage in Lightweight LLMs
AI总结：
本文提出一种 RLAIF 框架，利用大模型 AI 反馈训练奖励模型以提升轻量级 LLM 的 API 调用代码生成能力，在 Gorilla 数据集上可执行率提升 4.5%，且 780M 模型超越 7B 微调基线。
摘要：
来自 AI 反馈的强化学习（RLAIF）已在多个领域展现出显著潜力，包括减少大语言模型（LLM）输出中的危害、增强文本摘要和数学推理。本文提出一个 RLAIF 框架，用于提升轻量级（参数量小于 1B）LLM 的代码生成能力。我们特别关注需要编写恰当 API 调用的代码生成任务，由于 LLM 中众所周知的幻觉问题，这一任务具有挑战性。我们的框架通过专门的提示策略从更大的 LLM（如 GPT-3.5）中提取 AI 反馈，并利用这些数据训练奖励模型，以使较小 LLM 更好地对齐。我们在 Gorilla 数据集上进行实验，并通过多种指标（包括 AST、ROUGE 和 Code-BLEU）细致评估模型生成代码的质量，同时开发了一条流水线来准确计算其可执行率。我们的方法显著提升了微调 LLM 基线的性能，可执行率提高了 4.5%。值得注意的是，使用 RLAIF 训练的较小 LLM 模型（780M 参数）超越了参数量达 7B 的更大微调基线，代码可执行率高出 1.0%。
英文摘要：
Reinforcement Learning from AI Feedback (RLAIF) has demonstrated significant potential across various domains, including mitigating harm in LLM outputs, enhancing text summarization, and mathematical reasoning. This paper introduces an RLAIF framework for improving the code generation abilities of lightweight (<1B parameters) LLMs. We specifically focus on code generation tasks that require writing appropriate API calls, which is challenging due to the well-known issue of hallucination in LLMs. Our framework extracts AI feedback from a larger LLM (e.g., GPT-3.5) through a specialized prompting strategy and uses this data to train a reward model towards better alignment from smaller LLMs. We run our experiments on the Gorilla dataset and meticulously assess the quality of the model-generated code across various metrics, including AST, ROUGE, and Code-BLEU, and develop a pipeline to compute its executability rate accurately. Our approach significantly enhances the fine-tuned LLM baseline's performance, achieving a 4.5% improvement in executability rate. Notably, a smaller LLM model (780M parameters) trained with RLAIF surpasses a much larger fine-tuned baseline with 7B parameters, achieving a 1.0% higher code executability rate.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
