Solving complex or long-horizon problems often requires large language models (LLMs) to use external tools and operate over a significantly longer context window. New LLMs enable longer context windows and support tool calling capabilities. Prior works have focused mainly on evaluation of LLMs on long-context prompts, leaving agentic setup relatively unexplored, both from capability and safety perspectives. Our work addresses this gap. We find that LLM agents could be sensitive to length, type, and placement of the context, exhibiting unexpected and inconsistent shifts in task performance and in refusals to execute harmful requests. Models with 1M-2M token context windows show severe degradation already at 100K tokens, with performance drops exceeding 50\% for both benign and harmful tasks. Refusal rates shift unpredictably: GPT-4.1-nano increases from $\sim$5\% to $\sim$40\% while Grok 4 Fast decreases from $\sim$80\% to $\sim$10\% at 200K tokens. Our work shows potential safety issues with agents operating on longer context and opens additional questions on the current metrics and paradigm for evaluating LLM agent safety on long multi-step tasks. In particular, our results on LLM agents reveal a notable divergence in both capability and safety performance compared to prior evaluations of LLMs on similar criteria.
Deactivating Refusal Triggers: Understanding and Mitigating Overrefusal in Safety Alignment
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Johns Hopkins University（约翰霍普金斯大学）；Michigan State University（密歇根州立大学）；University of Mississippi（密苏里州立大学）
Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China（数据科学学院，香港中文大学（深圳））；Tencent AI Lab（腾讯人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）；Shenzhen Research Institute of Big Data, China（深圳大数据研究院）
Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibration
School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；CSIRO（澳大利亚联邦科学与工业研究组织）；Center of Excellence for Generative AI, King Abdullah University of Science and Technology（阿卜杜拉国王科技大学生成式人工智能卓越中心）
A Granular Study of Safety Pretraining under Model Abliteration
Data and Web Science Group, University of Mannheim（曼海姆大学数据与网络科学组）；Vision and AI Lab, Indian Institute of Science（印度科学院视觉与人工智能实验室）；Carnegie Mellon University（卡内基梅隆大学）；Max-Planck-Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息研究所，萨尔兰信息校园）
Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding
School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学信息科学与技术学院深圳校区）；School of Information, Central University of Finance and Economics（中央财经大学信息学院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Computing Technologies, RMIT University（皇家墨尔本理工大学计算技术学院）
DualEdit: Mitigating Safety Fallback in LLM Backdoor Editing via Affirmation-Refusal Regulation
University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Singapore Management University（新加坡管理大学）；MoE Key Lab of BIPC, University of Science and Technology of China（中国科学技术大学信息与通信技术国家重点实验室）
SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models
The University of Hong Kong (HKU)（香港大学）；Chinese Academy of Sciences (CAS)（中国科学院）；Information Engineering University (IEU)（信息工程大学）；Nanyang Technological University (NTU)（南洋理工大学）
Graph-Regularized Sparse Autoencoders for LLM Safety Steering
ELLIS Institute Tübingen（图宾根ELLIS研究所）；Max Planck Institute for Intelligent Systems（智能系统马克斯·普朗克研究所）；Intesa Sanpaolo（Intesa Sanpaolo公司）；University of Southern California（南加州大学）
