arXiv 2505.099252025-05-15cs.LGcs.AI

通过实时含噪人类反馈实现的强化交互式持续学习

Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback

School of Computer Science and Technology, East China Normal University（东华师范大学计算机科学与技术学院）；Shanghai Innovation Institute（上海创新研究院）；School of Computer Science, Fudan University（复旦大学计算机科学学院）

Yutao Yang, Jie Zhou, Junsong Li, Qianjun Pan, Bihao Zhan, Qin Chen, Xipeng Qiu, Liang He

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对实时流式人类反馈含噪且需动态更新的持续学习问题，提出RiCL框架，通过净化、偏好优化和抗噪对比学习提升性能。

摘要：

本文介绍了一种交互式持续学习范式，在该范式中，AI模型可从实时人类反馈中动态学习新技能，同时保留先前知识。该范式专门解决传统持续学习的两大局限：（1）利用流式、实时的人工标注数据进行动态模型更新，而非使用具有固定标签的静态数据集；（2）通过显式处理现实交互中常见的含噪反馈，打破标签干净的假设。为解决这些问题，我们提出RiCL，这是一个利用大型语言模型（LLM）从动态反馈中有效学习新技能的强化交互式持续学习框架。RiCL包含三个关键组件：时间一致性感知净化器，用于自动区分数据流中的干净样本与含噪样本；交互感知直接偏好优化策略，通过协调AI生成反馈与人类提供反馈，使模型行为与人类意图对齐；抗噪对比学习模块，通过利用数据内在关系捕获鲁棒表示，从而避免依赖可能不可靠的标签。在两个被真实噪声模式污染的基准数据集（FewRel和TACRED）上进行的大量实验表明，我们的RiCL方法显著优于现有最先进的在线持续学习与含噪标签学习方法的组合。

英文摘要：

This paper introduces an interactive continual learning paradigm where AI models dynamically learn new skills from real-time human feedback while retaining prior knowledge. This paradigm distinctively addresses two major limitations of traditional continual learning: (1) dynamic model updates using streaming, real-time human-annotated data, rather than static datasets with fixed labels, and (2) the assumption of clean labels, by explicitly handling the noisy feedback common in real-world interactions. To tackle these problems, we propose RiCL, a Reinforced interactive Continual Learning framework leveraging Large Language Models (LLMs) to learn new skills effectively from dynamic feedback. RiCL incorporates three key components: a temporal consistency-aware purifier to automatically discern clean from noisy samples in data streams; an interaction-aware direct preference optimization strategy to align model behavior with human intent by reconciling AI-generated and human-provided feedback; and a noise-resistant contrastive learning module that captures robust representations by exploiting inherent data relationships, thus avoiding reliance on potentially unreliable labels. Extensive experiments on two benchmark datasets (FewRel and TACRED), contaminated with realistic noise patterns, demonstrate that our RiCL approach substantially outperforms existing combinations of state-of-the-art online continual learning and noisy-label learning methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。