arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.04945cs.CLcs.AI
收藏

逐步精通:增强大型语言模型的软约束遵循能力

Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models

Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University(上海数据科学 key laboratory,计算机科学与人工智能学院,复旦大学);School of Data Science, Fudan University(数据科学学院,复旦大学);Ant Group(蚂蚁集团)

Qingyu Ren, Jie Zeng, Qianyu He, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, Fei Yu

AI总结:

针对大型语言模型软约束遵循能力不足的问题,提出自动构建数据集流水线,结合DPO与基于约束数量的课程学习范式,有效提升了模型的软约束遵循能力。

摘要:

对于大型语言模型(LLMs)而言,遵循涉及多重约束的指令至关重要。然而,增强LLMs遵循软约束的能力仍是一个未被探索的领域。为弥补这一空白,我们首先设计了一个流水线,以自动构建具有高质量输出的数据集。此外,为了充分利用数据构建过程中生成的正负样本,我们选择直接偏好优化(DPO)作为训练方法。进一步地,考虑到约束数量所体现的软约束难度,我们设计了一种基于约束数量的课程学习训练范式。我们通过实验评估了所提方法在提升LLMs软约束遵循能力方面的有效性,并分析了驱动性能提升的因素。数据集和代码已公开于https://github.com/Rainier-rq/FollowSoftConstraint。

英文摘要:

It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, it is an unexplored area to enhance LLMs' ability to follow soft constraints. To bridge the gap, we initially design a pipeline to construct datasets with high-quality outputs automatically. Additionally, to fully utilize the positive and negative samples generated during the data construction process, we choose Direct Preference Optimization (DPO) as the training method. Furthermore, taking into account the difficulty of soft constraints indicated by the number of constraints, we design a curriculum learning training paradigm based on the constraint quantity. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements.The datasets and code are publicly available at https://github.com/Rainier-rq/FollowSoftConstraint.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑