IOPO：通过输入-输出偏好优化增强大语言模型的复杂指令遵循能力
IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization
AI总结：
针对大语言模型复杂指令遵循能力不足及缺乏专用算法的问题，本文提出TRACE基准和IOPO输入-输出偏好优化方法，实验证明其显著优于SFT和DPO。
摘要：
在大语言模型（LLMs）领域，随着越来越多的智能体和应用利用LLMs进行构建，指令的复杂性正在迅速增加，模型准确遵循指令的能力变得至关重要。然而，一方面，现有的复杂指令评估数据数量有限；另一方面，目前缺乏专门的算法来提升复杂指令遵循能力。为此，本文引入了TRACE，一个用于提升和评估复杂指令遵循能力的基准，包含120K训练数据和1K评估数据。此外，我们提出了IOPO（Input-Output Preference Optimization，输入-输出偏好优化）对齐方法，该方法同时考虑输入和输出偏好对，使LLMs不仅能快速对齐响应偏好，还能细致地探索指令偏好。在域内和域外数据集上的大量实验证实了IOPO的有效性，与SFT和DPO相比，在域内数据上分别取得了8.15%和2.18%的提升，在域外数据上分别取得了6.29%和3.13%的提升。
英文摘要：
In the realm of large language models (LLMs), the ability of models to accurately follow instructions is paramount as more agents and applications leverage LLMs for construction, where the complexity of instructions are rapidly increasing. However, on the one hand, there is only a certain amount of complex instruction evaluation data; on the other hand, there are no dedicated algorithms to improve the ability to follow complex instructions. To this end, this paper introduces TRACE, a benchmark for improving and evaluating the complex instructionfollowing ability, which consists of 120K training data and 1K evaluation data. Furthermore, we propose IOPO (Input-Output Preference Optimization) alignment method which takes both input and output preference pairs into consideration, where LLMs not only rapidly align with response preferences but also meticulously explore the instruction preferences. Extensive experiments on both in-domain and outof-domain datasets confirm the effectiveness of IOPO, showing 8.15%, 2.18% improvements on in-domain data and 6.29%, 3.13% on outof-domain data compared to SFT and DPO respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
