IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉科学学院）；Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences（中国科学院重庆绿色智能技术研究院）；Zhejiang University（浙江大学）；Yale University（耶鲁大学）
We introduce IFIR, the first comprehensive benchmark designed to evaluate instruction-following information retrieval (IR) in expert domains. IFIR includes 2,426 high-quality examples and covers eight subsets across four specialized domains: finance, law, healthcare, and science literature. Each subset addresses one or more domain-specific retrieval tasks, replicating real-world scenarios where customized instructions are critical. IFIR enables a detailed analysis of instruction-following retrieval capabilities by incorporating instructions at different levels of complexity. We also propose a novel LLM-based evaluation method to provide a more precise and reliable assessment of model performance in following instructions. Through extensive experiments on 15 frontier retrieval models, including those based on LLMs, our results reveal that current models face significant challenges in effectively following complex, domain-specific instructions. We further provide in-depth analyses to highlight these limitations, offering valuable insights to guide future advancements in retriever development.
VCIFBench: Evaluating Complex Instruction Following for Video Understanding
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University（知识驱动人机智能工程研究中心，吉林大学）；International Center of Future Science, Jilin University（未来科学国际中心，吉林大学）
InFoBench: Evaluating Instruction Following Ability in Large Language Models
Tencent AI Lab, Seattle（腾讯人工智能实验室（西雅图））；University of Central Florida（中佛罗里达大学）；University of Georgia（佐治亚大学）；Emory University（埃默里大学）；Shanghai Jiao Tong University（上海交通大学）
Evaluating Evidence Grounding Under User Pressure in Instruction-Tuned Language Models
College of Information Sciences and Technology, Pennsylvania State University（宾夕法尼亚州立大学信息科学与技术学院）；Department of Computer Science, Old Dominion University（老 Dominion 大学计算机科学系）
S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models
Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanjing University（南京大学）；Shenzhen Loop Area Institute（深圳河套学院）；CentraleSupélec, Université Paris-Saclay（巴黎萨克雷大学CentraleSupélec分校）；National University of Singapore（新加坡国立大学）
BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions
Dept. of Comp. Sci. \& Tech., Institute for AI, Tsinghua University, Beijing, China；Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China
MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks
Northeastern University（东北大学）；Nanjing University of Science and Technology（南京理工大学）；University of Göttingen（哥廷根大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Beyond Ideal Instruction: A Comprehensive Framework for Evaluating LLMs in Realistic Interactions
Department of Data Science, City University of Hong Kong（香港城市大学数据科学系）；Hong Kong Institute of AI for Science, City University of Hong Kong（香港城市大学人工智能科学研究院）；Li Auto Inc.（Li汽车公司）；Beijing University of Posts and Telecommunications（北京邮电大学）；Independent Researcher（独立研究员）
Beyond Instruction Following: Evaluating Inferential Rule Following of Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Xiaohongshu Inc AI Lab, AIGility Cloud Innovation（小红书AI实验室）；Department of Computer Science and Engineering, Texas A&M University（德克萨斯农工大学计算机科学与工程学院）
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models
HKUST（香港科技大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；Ohio State University（俄亥俄州立大学）；AIWaves Inc.（AIWaves公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, Chinese Academy of Sciences（中国科学院人工智能学院）；Peking University（北京大学）；Beijing Foreign Studies University（北京外国语大学）；University of Manchester（曼彻斯特大学）；University of Waterloo（滑铁卢大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）
Evaluating and Achieving Controllable Code Completion in Code LLM
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）
