arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.08398cs.CLcs.AIcs.LG
收藏

揭示大型语言模型中的行为可塑性:基于令牌条件的视角

Revealing Behavioral Plasticity in Large Language Models: A Token-Conditional Perspective

Liyuan Mao, Le Yu, Jing Zhou, Chujie Zheng, Bowen Yu, Chang Gao, Shixuan Liu, An Yang, Weinan Zhang, JunYang Lin

AI总结:

本文提出ToCoRL框架,通过令牌条件强化学习使LLM在推理时实现稳定的行为适应,从而实现精确的行为控制。

摘要:

在本工作中,我们揭示大型语言模型(LLMs)具有内在的行为可塑性——类似于变色龙根据环境提示调整其颜色——可以通过令牌条件生成暴露,并通过强化学习稳定。具体而言,通过将生成过程条件化为精心选择的令牌前缀(这些前缀来自表现出期望行为的响应),LLMs能够在推理时间无缝地适应其行为模式(例如,从逐步推理切换到直接回答)而无需重新训练。基于这一洞察,我们提出了令牌条件强化学习(ToCoRL),一个原理性的框架,利用强化学习来内化这种变色龙般的可塑性,将短暂的推理时间适应转化为稳定且可学习的行为模式。ToCoRL通过令牌条件生成引导探索,并持续增强利用,使适当的行为得以涌现。广泛实验表明,ToCoRL能够在不降低能力的情况下实现精确的行为控制。值得注意的是,我们展示大型推理模型在复杂数学问题上表现强劲,但可以被有效适应以在事实性问答任务上表现出色,而这一能力此前受到逐步推理模式的阻碍。

英文摘要:

In this work, we reveal that Large Language Models (LLMs) possess intrinsic behavioral plasticity-akin to chameleons adapting their coloration to environmental cues-that can be exposed through token-conditional generation and stabilized via reinforcement learning. Specifically, by conditioning generation on carefully selected token prefixes sampled from responses exhibiting desired behaviors, LLMs seamlessly adapt their behavioral modes at inference time (e.g., switching from step-by-step reasoning to direct answering) without retraining. Based on this insight, we propose Token-Conditioned Reinforcement Learning (ToCoRL), a principled framework that leverages RL to internalize this chameleon-like plasticity, transforming transient inference-time adaptations into stable and learnable behavioral patterns. ToCoRL guides exploration with token-conditional generation and keep enhancing exploitation, enabling emergence of appropriate behaviors. Extensive experiments show that ToCoRL enables precise behavioral control without capability degradation. Notably, we show that large reasoning models, while performing strongly on complex mathematics, can be effectively adapted to excel at factual question answering, which was a capability previously hindered by their step-by-step reasoning patterns.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑