Large language models are increasingly deployed as specialized agents that plan, call tools, and take actions over extended horizons. Yet many existing evaluations assume a "clean interface" where dynamics are specified and stable, tools and sensors are reliable, and success is captured by a single explicit objective-often overestimating real-world readiness. In practice, agents face underspecified rules, unreliable signals, shifting environments, and implicit, multi-stakeholder goals. The challenge is therefore not just solving tasks, but adapting while solving: deciding what to trust, what is wanted, when to verify, and when to fall back or escalate. We stress-test deployment-relevant robustness under four operational circumstances: partial observability, dynamic environments, noisy signals, and dynamic agent state. We benchmark agentic LLMs in a grid-based game with a simple goal but long-horizon execution. Episodes violate clean-interface assumptions yet remain solvable, forcing agents to infer rules, pay for information, adapt to environmental and internal shifts, and act cautiously under noise. Across five state-of-the-art LLM agents, we find large gaps between nominal task-solving and deployment-like robustness. Performance generally degrades as grid size and horizon increase, but rankings are unstable: weaker models can beat stronger ones when strategy matches the uncertainty regime. Despite no explicit instruction, agents trade off completion, efficiency, and penalty avoidance, suggesting partial objective inference. Ablations and feature analyses reveal model-specific sensitivities and failure drivers, motivating work on verification, safe action selection, and objective inference under partial observability, noise, and non-stationarity.
HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, School of Artificial Intelligence, Beihang University(北京未来区块链与隐私计算先进创新中心,人工智能学院,北京航空航天大学);Tsinghua University(清华大学)
UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text
University of Maribor(马里博尔大学);University of Ljubljana(卢布尔雅那大学);Columbia University(哥伦比亚大学);Manisa Celal Bayar University(马尼萨杰拉尔·巴亚尔大学);University of Edinburgh(爱丁堡大学)
Large Language Model-based Human-Agent Collaboration for Complex Task Solving
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)
Enhancing Conversational Agents via Task-Oriented Adversarial Memory Adaptation
Xi’an Jiaotong University(西安交通大学);City University of Hong Kong(香港城市大学);University of Science and Technology of China(中国科学技术大学);Zhejiang University(浙江大学);Zhejiang University of Technology(浙江工业大学)