arXiv 2406.000372024-05-27cs.CLcs.AI

通过基于多视角用户偏好排序的反馈对齐LLM以进行编程问答

Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering

School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Hefei University of Technology（合肥工业大学）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；University of Toronto（多伦多大学）

Hongyu Yang, Liyang He, Min Hou, Shuanghong Shen, Rui Li, Jiahui Hou, Jianhui Ma, Junda Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出ALMupQA框架，通过多视角偏好排序对齐和检索增强上下文学习模块，结合新构建的StaCCQA数据集，有效提升了LLM在代码社区问答中满足用户多样化偏好和准确性的能力。

摘要：

代码社区问答（CCQA）旨在解决编程相关问题，从而提升软件工程和学术研究的生产力。基于人类反馈的强化学习（RLHF）的最新进展改变了大型语言模型（LLMs）的微调过程，使其能够生成紧密模仿人类行为的回答。因此，将LLMs与RLHF结合用于实际的CCQA应用已成为一个有前景的研究领域。与标准代码问答任务不同，CCQA包含多种可能的答案，且用户对每个回答有不同的偏好。此外，代码社区通常偏好新的API。这些挑战阻碍了LLMs生成满足CCQA任务中用户多样化偏好的回答。为了解决这些问题，我们提出了一个名为“通过基于多视角用户偏好排序的反馈对齐LLM以进行编程问答”（ALMupQA）的新框架，以创建以用户为中心的回答。我们的方法从多视角偏好排序对齐（MPRA）开始，它根据代码社区中答案的特征综合了多样的用户偏好。然后，我们引入了检索增强上下文学习（RIL）模块，通过从问题库中检索类似问题的回答来缓解答案过时的问题。由于高质量、多答案CCQA数据集的可用性有限，我们还从真实代码社区开发了一个名为StaCCQA的数据集。大量实验证明了ALMupQA框架在准确性和用户偏好方面的有效性。与基础模型相比，ALMupQA在BLEU上显示出近11%的提升，在BERTScore和CodeBERTScore上分别提高了20%和17.5%。

英文摘要：

Code Community Question Answering (CCQA) seeks to tackle programming-related issues, thereby boosting productivity in both software engineering and academic research. Recent advancements in Reinforcement Learning from Human Feedback (RLHF) have transformed the fine-tuning process of Large Language Models (LLMs) to produce responses that closely mimic human behavior. Leveraging LLMs with RLHF for practical CCQA applications has thus emerged as a promising area of study. Unlike standard code question-answering tasks, CCQA involves multiple possible answers, with varying user preferences for each response. Additionally, code communities often show a preference for new APIs. These challenges prevent LLMs from generating responses that cater to the diverse preferences of users in CCQA tasks. To address these issues, we propose a novel framework called Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering (ALMupQA) to create user-focused responses. Our approach starts with Multi-perspective Preference Ranking Alignment (MPRA), which synthesizes varied user preferences based on the characteristics of answers from code communities. We then introduce a Retrieval-augmented In-context Learning (RIL) module to mitigate the problem of outdated answers by retrieving responses to similar questions from a question bank. Due to the limited availability of high-quality, multi-answer CCQA datasets, we also developed a dataset named StaCCQA from real code communities. Extensive experiments demonstrated the effectiveness of the ALMupQA framework in terms of accuracy and user preference. Compared to the base model, ALMupQA showed nearly an 11% improvement in BLEU, with increases of 20% and 17.5% in BERTScore and CodeBERTScore, respectively.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。