arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.11541cs.CLcs.AI
收藏

MuSC:通过多粒度自对比训练改进复杂指令遵循

MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training

Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);Nanjing University(南京大学);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)

Hui Huang, Jiaheng Liu, Yancheng He, Shilong Li, Bing Xu, Conghui Zhu, Muyun Yang, Tiejun Zhao

AI总结:

本文提出多粒度自对比训练框架MuSC,通过指令分解重组构建约束感知偏好数据,并执行动态token级监督的偏好优化,在不依赖更强模型的情况下显著提升了LLMs的复杂与通用指令遵循能力。

摘要:

具有精细约束的复杂指令遵循对大型语言模型(LLMs)至关重要。尽管现有方法已构建用于复杂指令对齐的数据,但它们均依赖更高级的模型(尤其是GPT-4),限制了其应用。本文提出多粒度自对比训练(MuSC)框架,在不依赖更强模型的情况下改进复杂指令对齐。该方法在粗粒度和细粒度上均进行了操作:在粗粒度上,基于指令分解与重组构建约束感知偏好数据;在细粒度上,执行带有动态token级监督的token感知偏好优化。在开源模型上的评估结果表明,该方法在复杂和通用指令遵循基准上均取得显著提升,超越了以往的自对齐方法。

英文摘要:

Complex instruction-following with elaborate constraints is imperative for Large Language Models (LLMs). While existing methods have constructed data for complex instruction alignment, they all rely on a more advanced model, especially GPT-4, limiting their application. In this paper, we propose a Multi-granularity Self-Contrastive Training (MuSC) framework, to improve the complex instruction alignment without relying on a stronger model. Our method is conducted on both coarse and fine granularity. On coarse-granularity, we construct constraint-aware preference data based on instruction decomposition and recombination. On fine-granularity, we perform token-aware preference optimization with dynamic token-level supervision. Our method is evaluated on open-sourced models, and experiment results show our method achieves significant improvement on both complex and general instruction-following benchmarks, surpassing previous self-alignment methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑