Large language models (LLMs), initially developed for generative AI, are now evolving into agentic AI systems, which make decisions in complex, real-world contexts. Unfortunately, while their generative capabilities are well-documented, their decision-making processes remain poorly understood. This is particularly evident when testing targeted decision-making: for instance, how models handle exceptions, a critical and challenging aspect of decision-making made relevant by the inherent incompleteness of contracts. Here we demonstrate that LLMs, even ones that excel at reasoning, deviate significantly from human judgments because they adhere strictly to policies, even when such adherence is impractical, suboptimal, or even counterproductive. We then evaluate three approaches to tuning AI agents to handle exceptions: ethical framework prompting, chain-of-thought reasoning, and supervised fine-tuning. We find that while ethical framework prompting fails and chain-of-thought prompting provides only slight improvements, supervised fine-tuning - specifically with human explanations - yields markedly better results. Surprisingly, in our experiments, supervised fine-tuning even enabled models to generalize human-like decision-making to novel scenarios, demonstrating transfer learning of human-aligned decision-making across contexts. Furthermore, fine-tuning with explanations, not just labels, was critical for alignment, suggesting that aligning LLMs with human judgment requires explicit training on how decisions are made, not just which decisions are made. These findings highlight the need to address LLMs' shortcomings in handling exceptions in order to guide the development of agentic AI toward models that can effectively align with human judgment and simultaneously adapt to novel contexts.
MPI（马克斯·普朗克研究所）；ETH Zürich（苏黎世联邦理工学院）；MIT（麻省理工学院）；Harvard（哈佛大学）；IIT Kharagpur（印度理工学院卡拉格普尔分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Michigan（密歇根大学）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humans
Wuhan University（武汉大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Harbin Institute of Technology（哈尔滨工业大学）
Conversational Education at Scale: A Multi-LLM Agent Workflow for Procedural Learning and Pedagogic Quality Assessment
Vrije University of Amsterdam（阿姆斯特丹自由大学）；University College London（伦敦大学学院）；University of Amsterdam（阿姆斯特丹大学）；National Institute of Informatics（日本信息处理学会）；Shandong University（山东大学）；Guangzhou University（广州大学）
Can Large Language Models Predict the Outcome of Judicial Decisions?
Qatar University（卡塔尔大学）；OUC（中国海洋大学）；LJMU（利物浦约翰摩尔大学）；Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；Jordan University of Science and Technology（约旦科学技术大学）
