arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.06470cs.CLcs.AI
收藏

通过用户日志改进大型语言模型系统

Improve Large Language Model Systems with User Logs

Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)

Changyue Wang, Weihang Su, Qingyao Ai, Xingzhao Yue, Rui Zhang, Xiaojia Chang, Yiqun Liu

AI总结:

本文提出UNO框架,通过用户日志提炼规则和偏好对,利用查询反馈驱动聚类处理数据异质性,量化模型知识与日志数据间的认知差距,提升LLM系统性能。

摘要:

扩大训练数据和模型参数规模长期以来推动了大型语言模型(LLMs)的发展,但这一范式日益受到高质量数据稀缺和计算成本上升导致的边际效益递减的限制。因此,近期研究更加关注从真实世界部署中持续学习,其中用户交互日志提供了丰富的真人类反馈和过程知识。然而,从用户日志学习具有挑战性,因为它们是无结构和嘈杂的。传统的LLM系统往往难以区分有用的反馈信号与嘈杂的用户行为,且用户日志收集与模型优化之间的差异(例如,非策略优化问题)进一步加剧了这一问题。为此,我们提出UNO(用户日志驱动的优化),一个统一的框架,用于通过用户日志改进LLM系统(LLMsys)。UNO首先将日志提炼为半结构化的规则和偏好对,然后利用查询和反馈驱动的聚类来管理数据异质性,最后量化模型先验知识与日志数据之间的认知差距。这一评估指导LLMsys自适应地过滤掉嘈杂的反馈并构建不同模块,以处理从用户日志中提取的初级和反思性经验,从而提升未来的响应。广泛的实验表明,UNO在效果和效率上均达到最先进的水平,显著优于检索增强生成(RAG)和基于记忆的基线方法。我们已开源代码至https://github.com/bebr2/UNO。

英文摘要:

Scaling training data and model parameters has long driven progress in large language models (LLMs), but this paradigm is increasingly constrained by the scarcity of high-quality data and diminishing returns from rising computational costs. As a result, recent work is increasing the focus on continual learning from real-world deployment, where user interaction logs provide a rich source of authentic human feedback and procedural knowledge. However, learning from user logs is challenging due to their unstructured and noisy nature. Vanilla LLM systems often struggle to distinguish useful feedback signals from noisy user behavior, and the disparity between user log collection and model optimization (e.g., the off-policy optimization problem) further strengthens the problem. To this end, we propose UNO (User log-driveN Optimization), a unified framework for improving LLM systems (LLMsys) with user logs. UNO first distills logs into semi-structured rules and preference pairs, then employs query-and-feedback-driven clustering to manage data heterogeneity, and finally quantifies the cognitive gap between the model's prior knowledge and the log data. This assessment guides the LLMsys to adaptively filter out noisy feedback and construct different modules for primary and reflective experiences extracted from user logs, thereby improving future responses. Extensive experiments show that UNO achieves state-of-the-art effectiveness and efficiency, significantly outperforming Retrieval Augmented Generation (RAG) and memory-based baselines. We have open-sourced our code at https://github.com/bebr2/UNO .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑