Large language models (LLMs) have been widely deployed as autonomous agents capable of following user instructions and making decisions in real-world applications. Previous studies have made notable progress in benchmarking the instruction following capabilities of LLMs in general domains, with a primary focus on their inherent commonsense knowledge. Recently, LLMs have been increasingly deployed as domain-oriented agents, which rely on domain-oriented guidelines that may conflict with their commonsense knowledge. These guidelines exhibit two key characteristics: they consist of a wide range of domain-oriented rules and are subject to frequent updates. Despite these challenges, the absence of comprehensive benchmarks for evaluating the domain-oriented guideline following capabilities of LLMs presents a significant obstacle to their effective assessment and further development. In this paper, we introduce GuideBench, a comprehensive benchmark designed to evaluate guideline following performance of LLMs. GuideBench evaluates LLMs on three critical aspects: (i) adherence to diverse rules, (ii) robustness to rule updates, and (iii) alignment with human preferences. Experimental results on a range of LLMs indicate substantial opportunities for improving their ability to follow domain-oriented guidelines.
Benchmarking Clinical Decision Pathway Adherence in Large Language Models
Tongji University（同济大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tongji University School of Medicine（同济大学医学院）；Sun Yat-sen University（中山大学）；Fuzhou University（福州大学）；Shenzhen Loop Area Institute（深圳河套学院）
MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
Harvard University（哈佛大学）；MIT（麻省理工学院）；Cornell University（康奈尔大学）；Mayo Clinic（梅奥诊所）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；University of Virginia（弗吉尼亚大学）；Harvard Medical School（哈佛医学院）；Abaka AI；NYU（纽约大学）
Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking
Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机科学系）；Salesforce AI Research（Salesforce AI 研究院）；The Hong Kong University of Science and Technology（香港科学与技术大学）；National University of Singapore（新加坡国立大学）；The Education University of Hong Kong（香港教育大学）
A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations
Microsoft Research Asia（微软亚洲研究院）；Hong Kong University of Science and Technology（香港科技大学）；Peking University（北京大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese
Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））；Peking University（北京大学）；Mindverse AI；Alibaba Group（阿里巴巴集团）；Zhejiang University（浙江大学）；Zhejiang University of Technology（浙江工业大学）；MBZUAI；NIO（蔚来汽车）；HSBC（汇丰银行）；Harvard T.H. Chan School of Public Health（哈佛大学T.H. Chan公共卫生学院）
RESPClinBench: Benchmarking Multimodal Clinical Decision-Making and Longitudinal Disease Management in Respiratory Specialty Care
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Macau University of Science and Technology（澳门科技大学）；First Affiliated Hospital of Guangzhou Medical University（广州医科大学附属第一医院）；Guangzhou Institute of Respiratory Health（广州呼吸健康研究院）
Benchmarking large language models for biomedical natural language processing applications and recommendations
Yale University（耶鲁大学）；National Institutes of Health（美国国立卫生研究院）；University of Texas Health Science at Houston（休斯顿德克萨斯大学健康科学中心）；University of Minnesota（明尼苏达大学）
Rethinking Expert Trajectory Utilization in LLM Post-training for Mathematical Reasoning
Zhejiang University（浙江大学）；School of Engineering, Westlake University（西湖大学工程学院）；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖先进研究院技术研究所）；Huawei Noah’s Ark Lab（华为诺亚实验室）
EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction
Division of Computer Science, School of Electrical and Computer Engineering（计算机科学系，电气与计算机工程学院）；National Technical University of Athens（雅典技术大学）；Department of Humanities Social Sciences and Law, School of Applied Mathematical and Physical Sciences（人文社会科学与法律系，应用数学与物理科学学院）
SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents
National University of Singapore（新加坡国立大学）；Vietnam National University, Hanoi（越南国家大学河内）；University of Virginia, USA（弗吉尼亚大学）；Agency for Science, Technology and Research (A*STAR), Singapore（新加坡科技研究局）；Tufts University, USA（塔夫茨大学）；Virginia Tech, USA（弗吉尼亚理工大学）；Salesforce AI Research, USA（Salesforce AI 研究）
