While large language models (LLMs) have exhibited impressive instruction-following capabilities, it is still unclear whether and to what extent they can respond to explicit constraints that might be entailed in various instructions. As a significant aspect of LLM alignment, it is thus important to formulate such a specialized set of instructions as well as investigate the resulting behavior of LLMs. To address this vacancy, we propose a new benchmark CoDI-Eval to systematically and comprehensively evaluate LLMs' responses to instructions with various constraints. We construct a large collection of constraints-attributed instructions as a test suite focused on both generalization and coverage. Specifically, we advocate an instruction diversification process to synthesize diverse forms of constraint expression and also deliberate the candidate task taxonomy with even finer-grained sub-categories. Finally, we automate the entire evaluation process to facilitate further developments. Different from existing studies on controllable text generation, CoDI-Eval extends the scope to the prevalent instruction-following paradigm for the first time. We provide extensive evaluations of representative LLMs (e.g., ChatGPT, Vicuna) on CoDI-Eval, revealing their limitations in following instructions with specific constraints and there is still a significant gap between open-source and commercial closed-source LLMs. We believe this benchmark will facilitate research into improving the controllability of LLMs' responses to instructions. Our data and code are available at https://github.com/Xt-cyh/CoDI-Eval.
HACHIMI: Scalable and Controllable Student Persona Generation via Orchestrated Agents
East China Normal University(华东师范大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Shanghai Innovation Institute(上海创新研究院)
Benchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generation
University of Science and Technology of China(中国科学技术大学);City University of Hong Kong(香港城市大学);Beijing University of Posts and Telecommunications(北京邮电大学);Nanyang Technological University(南洋理工大学)
Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
Beijing University of Posts and Telecommunications(北京邮电大学);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Noumena AI
Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph
MBZUAI(穆罕默德·本·扎耶德人工智能大学);ETH Zürich(苏黎世联邦理工学院);Skoltech(斯科尔科沃科学技术学院);AIRI(人工智能研究所);HSE University(高等经济学院);FRC CSC RAS(俄罗斯科学院计算机科学控制问题科学中心);Nebius(Nebius公司);The University of Melbourne(墨尔本大学)
OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling
Shanghai University of Finance and Economics(上海财经大学);Booth School of Business, University of Chicago(芝加哥大学商学院);Antai School of Economics and Management, Shanghai Jiao Tong University(上海交通大学安泰经济管理学院)
DataGen: Unified Synthetic Dataset Generation via Large Language Models
University of Notre Dame(诺丁汉大学);Huazhong University of Science and Technology(华中科技大学);MBZUAI;University of Washington(华盛顿大学);Peking University(北京大学);University of Maryland, College Park(马里兰大学学院市分校);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Microsoft Research(微软研究院);Lehigh University(莱斯大学)
Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarking
The Hebrew University of Jerusalem(耶路撒冷希伯来大学);Allen Institute for AI(艾伦人工智能研究院);Bar-Ilan University(巴伊兰大学);Tel-Aviv University(特拉维夫大学);Stanford University(斯坦福大学)
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
AI Singapore, National University of Singapore(新加坡国立大学新加坡人工智能);School of Computing, National University of Singapore(新加坡国立大学计算学院);Institute of Advanced Intelligence and Computing, A*STAR(新加坡科技研究局先进智能与计算研究所)
LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K
Tsinghua University(清华大学);Infinigence-AI;Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Shanghai Jiao Tong University(上海交通大学)