arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.13352cs.CLcs.AIcs.MA
收藏

LLM-as-RNN: 一种用于内存更新和序列预测的循环语言模型

LLM-as-RNN: A Recurrent Language Model for Memory Updates and Sequence Prediction

Georgia Institute of Technology(佐治亚理工学院);Peking University(北京大学);Shandong University(山东大学);Huazhong University of Science and Technology(华中科技大学);UT Southwestern Medical Center(西南医学中心)

Yuxing Lu, J. Ben Tamo, Weichen Zhao, Nan Sun, Yishan Zhong, Wenqi Shi, Jinzhuo Wang, May D. Wang

AI总结:

LLM-as-RNN通过将冻结的LLM转化为循环预测器,利用自然语言记忆实现在线学习,有效提升序列预测精度并生成可解释的学习轨迹。

摘要:

大型语言模型是强大的序列预测器,但标准推断依赖于不可变的上下文历史。在生成步骤t出错后,模型缺乏可更新的记忆机制,无法改进步骤t+1的预测。我们提出了LLM-as-RNN,一种仅用于推断的框架,通过将隐藏状态表示为自然语言记忆,将冻结的LLM转换为循环预测器。该状态通过反馈驱动的文本重写在每个时间步更新,从而在不更新参数的情况下实现学习。在固定的token预算下,LLM-as-RNN能够纠正错误并保留任务相关模式,通过语言有效进行在线学习。我们在医疗、气象和金融三个连续基准上评估了该方法,覆盖Llama、Gemma和GPT模型家族。LLM-as-RNN在零样本、全历史和MemPrompt基线中显著优于,平均提高预测精度6.5%,同时生成可解释、可读的人类学习轨迹,而标准上下文积累则没有。

英文摘要:

Large language models are strong sequence predictors, yet standard inference relies on immutable context histories. After making an error at generation step t, the model lacks an updatable memory mechanism that improves predictions for step t+1. We propose LLM-as-RNN, an inference-only framework that turns a frozen LLM into a recurrent predictor by representing its hidden state as natural-language memory. This state, implemented as a structured system-prompt summary, is updated at each timestep via feedback-driven text rewrites, enabling learning without parameter updates. Under a fixed token budget, LLM-as-RNN corrects errors and retains task-relevant patterns, effectively performing online learning through language. We evaluate the method on three sequential benchmarks in healthcare, meteorology, and finance across Llama, Gemma, and GPT model families. LLM-as-RNN significantly outperforms zero-shot, full-history, and MemPrompt baselines, improving predictive accuracy by 6.5% on average, while producing interpretable, human-readable learning traces absent in standard context accumulation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑