Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Xiaohongshu Inc AI Lab, AIGility Cloud Innovation（小红书AI实验室）；Department of Computer Science and Engineering, Texas A&M University（德克萨斯农工大学计算机科学与工程学院）
Wangtao Sun, Chenxiang Zhang, XueYou Zhang, Xuanqing Yu, Ziyang Huang, Pei Chen, Haotian Xu, Shizhu He, Jun Zhao, Kang Liu
Although Large Language Models (LLMs) have demonstrated strong ability, they are further supposed to be controlled and guided by in real-world scenarios to be safe, accurate, and intelligent. This demands the possession of capability of LLMs. However, no prior work has made a clear evaluation of the inferential rule-following capability of LLMs. Previous studies that try to evaluate the inferential rule-following capability of LLMs fail to distinguish the inferential rule-following scenarios from the instruction-following scenarios. Therefore, this paper first clarifies the concept of inferential rule-following and proposes a comprehensive benchmark, RuleBench, to evaluate a diversified range of inferential rule-following abilities. Our experimental results on a variety of LLMs show that they are still limited in following rules. Our analysis based on the evaluation results provides insights into the improvements for LLMs toward a better inferential rule-following intelligent agent. We further propose Inferential Rule-Following Tuning (IRFT). The experimental results show that through IRFT, LLMs can learn abstract rule-following abilities from purely synthetic data and then generalize to RuleBench. The data and code can be found at: https://anonymous.4open.science/r/llm-rule-following-B3E3/
InFoBench: Evaluating Instruction Following Ability in Large Language Models
Tencent AI Lab, Seattle（腾讯人工智能实验室（西雅图））；University of Central Florida（中佛罗里达大学）；University of Georgia（佐治亚大学）；Emory University（埃默里大学）；Shanghai Jiao Tong University（上海交通大学）
In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search
University of Southern California（南加州大学）；Ohio State University（俄亥俄州立大学）；Fudan University（复旦大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Pittsburgh（匹兹堡大学）；University of Washington（华盛顿大学）；Cornell University（康奈尔大学）
VCIFBench: Evaluating Complex Instruction Following for Video Understanding
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University（知识驱动人机智能工程研究中心，吉林大学）；International Center of Future Science, Jilin University（未来科学国际中心，吉林大学）
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models
HKUST（香港科技大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；Ohio State University（俄亥俄州立大学）；AIWaves Inc.（AIWaves公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, Chinese Academy of Sciences（中国科学院人工智能学院）；Peking University（北京大学）；Beijing Foreign Studies University（北京外国语大学）；University of Manchester（曼彻斯特大学）；University of Waterloo（滑铁卢大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）
ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models
CoLing Lab, Department of Philology, Literature, and Linguistics, University of Pisa, Italy（皮尔森大学哲学、文学与语言学系协作语言实验室）；Department of Informatics, University of Pisa, Italy（皮尔森大学信息学系）；Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University（香港理工大学中文与双语研究系）
S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；Shenzhen Loop Area Institute（深圳河套学院）；CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学CentraleSupélec分校）；National University of Singapore（新加坡国立大学）
BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions
Dept. of Comp. Sci. \& Tech., Institute for AI, Tsinghua University, Beijing, China；Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
