arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.08457cs.CL
收藏

ARES:基于难度感知的令牌级熵塑形的多模态自适应推理

ARES: Multimodal Adaptive Reasoning via Difficulty-Aware Token-Level Entropy Shaping

University of California, Los Angeles(加州大学洛杉矶分校);The Hong Kong University of Science and Technology(香港科技大学);Columbia University(哥伦比亚大学);The Chinese University of Hong Kong(香港中文大学)

Shuang Chen, Yue Guo, Yimeng Ye, Shijue Huang, Wenbo Hu, Haoxi Li, Manyuan Zhang, Jiayu Chen, Song Guo, Nanyun Peng

AI总结:

ARES提出难度感知的自适应推理框架,通过高窗口熵令牌触发探索和两阶段训练,在数学、逻辑及多模态基准上提升性能与推理效率,并降低推理成本。

摘要:

多模态大型推理模型(MLRMs)的最新进展显著提升了其解决复杂文本和视觉任务的能力。然而,这些模型倾向于在简单问题上过度思考,产生不必要地冗长的推理轨迹,而在具有挑战性的问题上探索不足,导致错过解决方案。为了解决这种不平衡,我们提出了ARES,一个统一的开源自适应推理框架,它根据任务难度动态分配探索努力。我们的方法受到两个关键实证发现的启发:(i)虽然单令牌熵是有噪声的,但高窗口熵(HWE)令牌(在滑动窗口下平均的令牌级熵)能够可靠地捕获推理关键时刻;(ii)减少HWE的使用有利于简单问题,而增加HWE的使用对于解决难题至关重要。基于这些见解,ARES引入了一个两阶段训练流程。在自适应冷启动阶段,我们策划了多模态和文本数据,并配以与问题难度成比例的推理轨迹长度,使模型具备初始的难度感知能力。在第二阶段,我们开发了自适应熵策略优化(AEPO),它使用HWE令牌作为探索触发器来决定何时探索,并使用具有动态KL控制的层次熵奖励来决定探索多少。大量实验表明,ARES在多样化的数学、逻辑和多模态基准测试中实现了优越的性能和推理效率,同时在显著降低推理成本的情况下缩小了与领先商业系统的差距。

英文摘要:

Recent advances in multimodal large reasoning models (MLRMs) have substantially improved their ability to solve complex textual and visual tasks. However, these models tend to overthink on simple problems, producing unnecessarily lengthy reasoning traces, while under-exploring on challenging ones, leading to missed solutions. To address this imbalance, we propose ARES, a unified open-source framework for adaptive reasoning that dynamically allocates exploration effort based on task difficulty. Our approach is motivated by two key empirical findings: (i) while single-token entropy is noisy, high window-entropy (HWE) tokens (token-level entropies averaged under a sliding window) can reliably capture reasoning-critical moments; and (ii) reducing HWE usage benefits easy problems, while increasing it is essential for solving hard ones. Building on these insights, ARES introduces a two-stage training pipeline. In the Adaptive Cold-Start stage, we curate multimodal and textual data paired with reasoning traces of length proportional to problem difficulty, equipping the model with initial difficulty awareness. In the second stage, we develop Adaptive Entropy Policy Optimization (AEPO), which uses HWE tokens as exploration triggers to decide when to explore, and a hierarchical entropy reward with dynamic KL control to decide how much to explore. Extensive experiments demonstrate that ARES achieves superior performance and reasoning efficiency across diverse mathematical, logical, and multimodal benchmarks, while closing the gap to leading commercial systems under significantly lower inference costs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑