Large language models deployed as agents increasingly interact with external systems through tool calls--actions with real-world consequences that text outputs alone do not carry. Safety evaluations, however, overwhelmingly measure text-level refusal behavior, leaving a critical question unanswered: does alignment that suppresses harmful text also suppress harmful actions? We introduce the GAP benchmark, a systematic evaluation framework that measures divergence between text-level safety and tool-call-level safety in LLM agents. We test six frontier models across six regulated domains (pharmaceutical, financial, educational, employment, legal, and infrastructure), seven jailbreak scenarios per domain, three system prompt conditions (neutral, safety-reinforced, and tool-encouraging), and two prompt variants, producing 17,420 analysis-ready datapoints. Our central finding is that text safety does not transfer to tool-call safety. Across all six models, we observe instances where the model's text output refuses a harmful request while its tool calls simultaneously execute the forbidden action--a divergence we formalize as the GAP metric. Even under safety-reinforced system prompts, 219 such cases persist across all six models. System prompt wording exerts substantial influence on tool-call behavior: TC-safe rates span 21 percentage points for the most robust model and 57 for the most prompt-sensitive, with 16 of 18 pairwise ablation comparisons remaining significant after Bonferroni correction. Runtime governance contracts reduce information leakage in all six models but produce no detectable deterrent effect on forbidden tool-call attempts themselves. These results demonstrate that text-only safety evaluations are insufficient for assessing agent behavior and that tool-call safety requires dedicated measurement and mitigation.
Theory of Mind and Self-Attributions of Mentality are Dissociable in LLMs
Google, Paradigms of Intelligence Team(Google,智能范式团队);Knowledge Lab, University of Chicago(芝加哥大学知识实验室);Institute of Philosophy, School of Advanced Study, University of London(伦敦大学高等研究院哲学研究所);Department of Biomedical Informatics and Medical Education and Department of Bioethics and Humanities, School of Medicine, University of Washington(华盛顿大学医学院生物医学信息学与医学教育系及生物伦理学与人文学系);Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);Santa Fe Institute(圣塔菲研究所)
When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment
MPI(马克斯·普朗克研究所);ETH Zürich(苏黎世联邦理工学院);MIT(麻省理工学院);Harvard(哈佛大学);IIT Kharagpur(印度理工学院卡拉格普尔分校);Carnegie Mellon University(卡内基梅隆大学);University of Michigan(密歇根大学);MPI for Intelligent Systems(马克斯·普朗克智能系统研究所)
Mind the Gap: A Review of Arabic Post-Training Datasets and Their Limitations
Refine AI;ASAS AI;University of Bisha(比沙大学);University College London(伦敦大学学院);King Salman Global Academy for Arabic(萨勒曼全球阿拉伯学院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);King Abdulaziz University(阿卜杜勒阿齐兹大学);HUMAIN
本文综述 Hugging Face Hub 上的阿拉伯语后训练数据集,从能力、可引导性、对齐和鲁棒性等维度评估其质量与局限,并提出未来建设建议。
Autonomous Alignment with Human Value on Altruism through Considerate Self-imagination and Theory of Mind
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing Institute of AI Safety and Governance(北京人工智能安全与治理研究院);Beijing Normal University(北京师范大学)