arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.00690cs.CL
收藏

在多样化指令下对大型语言模型可控生成的基准测试

Benchmarking Large Language Models on Controllable Generation under Diversified Instructions

University of Science and Technology of China(中国科学技术大学);Beijing University of Posts and Telecommunications(北京邮电大学)

Yihan Chen, Benfeng Xu, Quan Wang, Yi Liu, Zhendong Mao

AI总结:

本文提出新基准CoDI-Eval,系统评估LLMs在多样化指令和显式约束下的响应能力,揭示现有模型局限性及开源与闭源模型的显著差距。

摘要:

尽管大型语言模型(LLMs)展现出了令人印象深刻的指令遵循能力,但目前仍不清楚它们是否以及在多大程度上能够响应各种指令中可能包含的显式约束。作为LLM对齐的一个重要方面,因此制定这样一个专门的指令集并研究LLM由此产生的行为显得很重要。为了填补这一空白,我们提出了一种新的基准测试CoDI-Eval,以系统且全面地评估LLMs对带有各种约束的指令的响应。我们构建了一个包含大量约束属性指令的集合作为测试套件,重点关注泛化能力和覆盖范围。具体而言,我们提倡一种指令多样化过程来合成多种形式的约束表达,并考虑具有更细粒度子类别的候选任务分类法。最后,我们实现了整个评估过程的自动化,以促进后续的发展。与现有的可控文本生成研究不同,CoDI-Eval首次将范围扩展到了流行的指令遵循范式。我们在CoDI-Eval上对代表性的LLMs(例如ChatGPT、Vicuna)进行了广泛评估,揭示了它们在遵循带有特定约束的指令方面的局限性,并且开源LLMs与商业闭源LLMs之间仍存在显著差距。我们相信该基准测试将促进关于提高LLMs对指令响应的可控性的研究。我们的数据和代码可在https://github.com/Xt-cyh/CoDI-Eval获取。

英文摘要:

While large language models (LLMs) have exhibited impressive instruction-following capabilities, it is still unclear whether and to what extent they can respond to explicit constraints that might be entailed in various instructions. As a significant aspect of LLM alignment, it is thus important to formulate such a specialized set of instructions as well as investigate the resulting behavior of LLMs. To address this vacancy, we propose a new benchmark CoDI-Eval to systematically and comprehensively evaluate LLMs' responses to instructions with various constraints. We construct a large collection of constraints-attributed instructions as a test suite focused on both generalization and coverage. Specifically, we advocate an instruction diversification process to synthesize diverse forms of constraint expression and also deliberate the candidate task taxonomy with even finer-grained sub-categories. Finally, we automate the entire evaluation process to facilitate further developments. Different from existing studies on controllable text generation, CoDI-Eval extends the scope to the prevalent instruction-following paradigm for the first time. We provide extensive evaluations of representative LLMs (e.g., ChatGPT, Vicuna) on CoDI-Eval, revealing their limitations in following instructions with specific constraints and there is still a significant gap between open-source and commercial closed-source LLMs. We believe this benchmark will facilitate research into improving the controllability of LLMs' responses to instructions. Our data and code are available at https://github.com/Xt-cyh/CoDI-Eval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑