arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.21754cs.AI
收藏

基于语言的试错在经验时代落后了

Language-based Trial and Error Falls Behind in the Era of Experience

University of Science and Technology of China(中国科学技术大学)

Haoyu Wang, Guozheng Ma, Shugang Cui, Yilun Kong, Haotian Luo, Li Shen, Mengya Gao, Yichao Wu, Xiaogang Wang, Dacheng Tao

AI总结:

针对LLM在非语言环境中探索成本高的问题,提出SCOUT框架,用轻量级模型探索环境,通过SFT和RL激活LLM的世界知识,显著提升性能并降低计算开销。

摘要:

尽管大型语言模型(LLM)在基于语言的智能体任务中表现出色,但它们对未见过的非语言环境(例如符号或空间任务)的适用性仍然有限。先前的工作将这种性能差距归因于预训练分布与测试分布之间的不匹配。在这项工作中,我们证明了主要瓶颈是探索的过高成本:掌握这些任务需要大量的试错,这对于在高维语义空间中运行的参数庞大的LLM来说在计算上是不可持续的。为了解决这个问题,我们提出了SCOUT(子规模协作处理未见任务),一种将探索与利用解耦的新框架。我们使用轻量级的“侦察兵”(例如小型MLP)以远超LLM的速度和规模探测环境动态。收集到的轨迹用于通过监督微调(SFT)引导LLM,然后通过多轮强化学习(RL)激活其潜在的世界知识。实验表明,SCOUT使Qwen2.5-3B-Instruct模型达到了0.86的平均得分,显著优于包括Gemini-2.5-Pro(0.60)在内的专有模型,同时节省了约60%的GPU小时消耗。

英文摘要:

While Large Language Models (LLMs) excel in language-based agentic tasks, their applicability to unseen, nonlinguistic environments (e.g., symbolic or spatial tasks) remains limited. Previous work attributes this performance gap to the mismatch between the pretraining distribution and the testing distribution. In this work, we demonstrate the primary bottleneck is the prohibitive cost of exploration: mastering these tasks requires extensive trial-and-error, which is computationally unsustainable for parameter-heavy LLMs operating in a high dimensional semantic space. To address this, we propose SCOUT (Sub-Scale Collaboration On Unseen Tasks), a novel framework that decouples exploration from exploitation. We employ lightweight "scouts" (e.g., small MLPs) to probe environmental dynamics at a speed and scale far exceeding LLMs. The collected trajectories are utilized to bootstrap the LLM via Supervised Fine-Tuning (SFT), followed by multi-turn Reinforcement Learning (RL) to activate its latent world knowledge. Empirically, SCOUT enables a Qwen2.5-3B-Instruct model to achieve an average score of 0.86, significantly outperforming proprietary models, including Gemini-2.5-Pro (0.60), while saving about 60% GPU hours consumption.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑