Instructions Are Evidence, Not Objectives: Measuring Outcome Alignment of Agents
College of AI, Tsinghua University(清华大学人工智能学院);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Institute for AI, Tsinghua University(清华大学人工智能研究院)
Kaiming Liu, Yuxuan Liu, Jinrui Ju, Minyuan Ruan, Xuanyu Lei, Peng Li, Yang Liu
Large language model agents can follow complex human instructions to solve real-world tasks. In practice, however, initial instructions are often incomplete. In our analysis of 100 user-agent trajectories from SWE-chat, 56\% of initial instructions omit requirements that later emerge as necessary, and 52\% of the useful follow-up requirements could have been inferred from the project. A user states requirements from their own understanding of the task, and these \textbf{stated requirements} often differ from the \textbf{true requirements} that completing the task imposes. An agent should therefore treat the instruction as evidence rather than as the objective, inferring the true requirements from the instruction and the environment and satisfying them even when the user is not yet aware of all of them. We call this ability \textbf{outcome alignment}. In this work, we construct \textbf{TiGanEval}, a benchmark of software engineering and research tasks. In every task the user genuinely lacks part of the requirements, each requirement stays discoverable from the code and runtime behaviour of the environment, and requirements the agent infers are separated from those the user adds after observing delivered work. These properties together make outcome alignment measurable. Evaluating eight frontier models including GPT-5.6-Sol, we find outcome alignment broadly missing: the models lose 53 to 67 points of success rate when moving from the full specification to the partial instruction, and consequence feedback across four submissions does not close the gap. Furthermore, a plan-mode harness cannot compensate for the missing ability of the model itself, which places outcome alignment among the abilities the next generation of models should be developed for.
Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
Universität Bielefeld(比勒菲尔德大学);Ruhr-Universität Bochum(波鸿鲁尔大学);ELLIS Institute(ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Tübingen AI Center(图宾根人工智能中心)
Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Technische Universität Berlin(柏林技术大学);IIIT Hyderabad(海得拉巴国家理工学院);Univ of Maryland(马里兰大学);Rice Univ(Rice 大学);Univ of Wisconsin - Madison(威斯康星大学麦迪逊分校);Spector Inc(Spector 公司);Microsoft(微软公司)
Can Segmentation Models Understand the World? Towards Proactive Affordance Reasoning via Visual Chain-of-Thought
Northwestern University(西北大学);Northeastern University(东北大学);South China University of Technology(华南理工大学);Hong Kong Baptist University(香港 Baptist大学);Beijing Normal - Hong Kong Baptist University(北京师范大学-香港 Baptist大学)
SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligence
Shanghai AI Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);The Chinese University of Hong Kong(香港中文大学);University of Sydney(悉尼大学);Fudan University(复旦大学);Tsinghua University(清华大学);Beihang University(北航大学)
Measuring Outcomes in Healthcare Economics using Artificial Intelligence: with Application to Resource Management
George Mason University(乔治梅森大学);Virginia Polytechnic Institute and State University(弗吉尼亚理工学院暨州立大学);Brigham and Women’s Hospital, Harvard Medical School(哈佛医学院布莱根妇女医院)
Learning to Follow Instructions in Text-Based Games
University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(矢量人工智能研究所);Schwartz Reisman Institute for Technology and Society(施瓦茨·赖斯曼技术与社会研究所)