While Large Language Models (LLMs) can exhibit impressive proficiency in isolated, short-term tasks, they often fail to maintain coherent performance over longer time horizons. In this paper, we present Vending-Bench, a simulated environment designed to specifically test an LLM-based agent's ability to manage a straightforward, long-running business scenario: operating a vending machine. Agents must balance inventories, place orders, set prices, and handle daily fees - tasks that are each simple but collectively, over long horizons (>20M tokens per run) stress an LLM's capacity for sustained, coherent decision-making. Our experiments reveal high variance in performance across multiple LLMs: Claude 3.5 Sonnet and o3-mini manage the machine well in most runs and turn a profit, but all models have runs that derail, either through misinterpreting delivery schedules, forgetting orders, or descending into tangential "meltdown" loops from which they rarely recover. We find no clear correlation between failures and the point at which the model's context window becomes full, suggesting that these breakdowns do not stem from memory limits. Apart from highlighting the high variance in performance over long time horizons, Vending-Bench also tests models' ability to acquire capital, a necessity in many hypothetical dangerous AI scenarios. We hope the benchmark can help in preparing for the advent of stronger AI systems.
When Stories Evolve: Benchmarking LLM Storytelling Across Agent Architectures in Open-Ended World Simulations
The University of Hong Kong(香港大学);Peking University(北京大学);Tsinghua University(清华大学);Beijing Institute of Mathematical Sciences and Applications (BIMSA)(北京数学科学与应用研究院)
School of Computer Science and Artificial Intelligence(计算机科学与人工智能学院);ETH Zürich(苏黎世联邦理工学院);School of Remote Sensing and Information Engineering(遥感与信息工程学院)
Distilling Temporal Coherence into 2D Networks for Transrectal Ultrasound Prostate Video Segmentation
Interdisciplinary Program in Bioengineering, Seoul National University(首尔大学生物工程跨学科项目);Department of Transdisciplinary Medicine, Seoul National University Hospital(首尔大学医院跨学科医学系);Department of Artificial Intelligence, Yonsei University(延世大学人工智能系);Department of Radiology, Seoul National University Hospital(首尔大学医院放射科)