Large language models (LLMs) are increasingly capable of carrying out long-running, real-world tasks. However, as the amount of context grows, their reliability often deteriorates, a phenomenon known as "context rot". Existing long-context benchmarks primarily focus on single-step settings that evaluate a model's ability to retrieve information from a long snippet. In realistic scenarios, however, LLMs often need to act as agents that explore environments, follow instructions and plans, extract useful information, and predict correct actions under a dynamically growing context. To assess language agents in such settings, we introduce LOCA-bench (a benchmark for LOng-Context Agents). Given a task prompt, LOCA-bench leverages automated and scalable control of environment states to regulate the agent's context length. This design enables LOCA-bench to extend the context length potentially to infinity in a controlled way while keeping the underlying task semantics fixed. LOCA-bench evaluates language agents as a combination of models and scaffolds, including various context management strategies. While agent performance generally degrades as the environment states grow more complex, advanced context management techniques can substantially improve the overall success rate. We open-source LOCA-bench to provide a platform for evaluating models and scaffolds in long-context, agentic scenarios: https://github.com/hkust-nlp/LOCA-bench
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Idiap Research Institute(Idiap研究 institute);EPFL(瑞士联邦理工学院);School of Computer Science, University of Sheffield(谢菲尔德大学计算机科学学院);University of Manchester(曼彻斯特大学);National Biomarker Centre, CRUK Manchester Institute(国家生物标志物中心,CRUK曼彻斯特研究所)
GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning
School of Cyberspace Science and Technology, Beijing Institute of Technology(航天信息科学技术学院,北京理工大学);School of Computer Science, The University of Auckland(计算机科学学院,奥克兰大学)
OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai AI Lab(上海人工智能实验室);Beihang University(北京航空航天大学);Nanyang Technological University(南洋理工大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Chinese University of Hong Kong(香港中文大学)
REACT: Representation Extraction And Controllable Tuning to Overcome Overfitting in LLM Knowledge Editing
NLPR, MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所国家信息处理实验室、人工智能研究所);Cuiying Honors College, Lanzhou University(兰州大学萃英荣誉学院);Department of Mathematics, The Chinese University of Hong Kong(香港中文大学数学系);Tencent(腾讯)
LLM Driven Design of Continuous Optimization Problems with Controllable High-level Properties
Machine Learning and Optimization, Paderborn University, Germany(机器学习与优化,帕德博恩大学,德国);Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands(莱顿先进计算机科学研究所,莱顿大学,荷兰);Data Management and Biometrics, University of Twente, The Netherlands(数据管理与生物特征,埃因霍温大学,荷兰)
MUSES: 3D-Controllable Image Generation via Multi-Modal Agent Collaboration
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学)