我们引入了Pencil Puzzle Bench，这是一个通过笔算谜题评估大型语言模型推理能力的框架，笔算谜题是一类与NP完全问题密切相关的约束满足问题，具有确定性的分步验证。从包含62,231个谜题的数据库中，涵盖94种类型且已验证唯一解的谜题中，我们选取了300个谜题，覆盖20种类型，并在两种模式下评估了来自11家提供商的51个模型：直接提问（单次）和代理（多轮带迭代验证）。我们基准的突出特点是，每个中间棋盘状态都可以根据特定类型的约束进行检查，将错误定位到确切违反的规则，为密集的每步奖励信号提供了基础设施，用于过程监督和强化学习。我们的评估揭示了两种不同的能力轴：（1）推理努力扩展，其中GPT-5.2从无推理提升81倍，从无推理到最大努力；（2）代理迭代，其中Claude Opus 4.6通过迭代检查从0.3%提升到30.0%，而GPT-5.2@xhigh从20.2%提升到56.0%。代理尝试的中位数为29轮，持续17分钟，最长超过1,221轮和14.3小时——对长上下文利用的严苛测试，不仅仅是推理。
英文摘要：
We introduce Pencil Puzzle Bench, a framework for evaluating large language model reasoning through pencil puzzles, a family of constraint-satisfaction problems closely related to NP-complete problems, with deterministic, step-level verification. From a database of 62,231 puzzles across 94 varieties with verified unique solutions, we select a benchmark of 300 puzzles spanning 20 varieties and evaluate 51 models from 11 providers in two modes: direct ask (single-shot) and agentic (multi-turn with iterative verification). A key differentiator of our benchmark is that every intermediate board state can be checked against variety-specific constraints, localizing errors to the exact rule violated, providing the infrastructure for dense, per-move reward signals for process supervision and reinforcement learning.
Our evaluation reveals two distinct axes of capability: (1) reasoning effort scaling, where GPT-5.2 improves 81x from no reasoning to maximum effort; and (2) agentic iteration, where Claude Opus 4.6 rises from 0.3% to 30.0% through iterative checking, while GPT-5.2@xhigh improves from 20.2% to 56.0%. Agentic attempts span a median of 29 turns over 17 minutes, with the longest exceeding 1,221 turns and 14.3 hours - a demanding test of long-context utilization, not just reasoning.
VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora
Institute of Information Engineering, CAS（中国科学院信息工程研究所）；School of Cyber Security, UCAS（中国科学技术大学网络安全学院）；Amap, Alibaba Group（阿里巴巴集团阿地图）；NLPR & MAIS, Institute of Automation, CAS（中国科学院自动化研究所神经信息处理实验室及机器智能研究所）；School of Artificial Intelligence, UCAS（中国科学技术大学人工智能学院）
East China Normal University（华东师范大学）；Huawei Singapore Research Center（华为新加坡研究中心）；Huawei Technologies Co., Ltd.（华为技术有限公司）
Formally verifying Deep Reinforcement Learning (DRL) systems is a challenging task due to the dynamic continuity of system behaviors and the black-box feature of embedded neural networks. In this paper, we propose a novel abstraction-based approach to train DRL systems on finite abstract domains instead of concrete system states. It yields neural networks whose input states are finite, making hosting DRL systems directly verifiable using model checking techniques. Our approach is orthogonal to existing DRL algorithms and off-the-shelf model checkers. We implement a resulting prototype training…
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Huawei（华为）
PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents
Tencent Hunyuan（腾讯文英）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院（ Gallagher 学院））
FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning
MBZUAI；Syllogia；The University of Tokyo（东京大学）；The Fin AI（Fin AI）；Cornell University（康奈尔大学）；The University of Melbourne（墨尔本大学）；IIT Delhi（德里理工学院）；The University of Manchester（曼彻斯特大学）
GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning
Central South University（中南大学）；Jiangxi Normal University（江西师范大学）；Hohai University（河海大学）；Renmin University of China（中国人民大学）；The Chinese University of Hong Kong（香港中文大学）
A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education（教育部多媒体可信感知与高效计算重点实验室）；Institute of Artificial Intelligence（人工智能研究院）
