Our goal is a teachable reasoning system for question-answering (QA), where a user can interact with faithful answer explanations, and correct its errors so that the system improves over time. Our approach is to augment a QA model with a dynamic memory of user feedback, containing user-supplied corrections to erroneous model beliefs that users identify during interaction. Retrievals from memory are used as additional context for QA, to help avoid previous mistakes in similar new situations - a novel application of memory-based continuous learning. With simulated feedback, we find that our system (called TeachMe) continually improves with time, and without model retraining, requiring feedback on only 25% of training examples to reach within 1% of the upper-bound (feedback on all examples). Similarly, in experiments with real users, we observe a similar trend, with performance improving by over 15% on a hidden test set after teaching. This suggests new opportunities for using frozen language models in an interactive setting where users can inspect, debug, and correct the model's beliefs, leading to improved system's performance over time.
Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback
School of Computer Science and Technology, East China Normal University(东华师范大学计算机科学与技术学院);Shanghai Innovation Institute(上海创新研究院);School of Computer Science, Fudan University(复旦大学计算机科学学院)
NEXUS: Learning Safety Constraints and Action Models for Safe and Robust Embodied Planning
School of Artificial Intelligence and Automation(人工智能与自动化学院);Huazhong University of Science and Technology(华中科技大学);School of Software(软件学院);Henan University(河南大学);Institute of Information Engineering(信息工程研究所);Chinese Academy of Sciences(中国科学院);School of Cyberspace Security(网络空间安全学院);University of the Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学)
Making Language Models Better Tool Learners with Execution Feedback
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-Ant Group Joint Research Center for Knowledge Graphs, Zhejiang University(浙江大学-蚂蚁集团知识图谱联合研究中心);ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University(浙江大学杭州国际科技创新中心);Alibaba Group(阿里巴巴集团)
SafeCoEvo: Co-Evolving Safety Harnesses and Guards for LLM Agents at Test-Time
East China Normal University(华东师范大学);Shanghai Innovation Institute(上海创新研究院);Xiamen University(厦门大学);Harbin Institute of Technology(哈尔滨工业大学);University College London(伦敦大学学院);Huawei Noah’s Ark Lab, UK(华为诺亚方舟实验室(英国));MemoraX AI
COPR: Continual Learning Human Preference through Optimal Policy Regularization
Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);King’s College London(伦敦国王学院);National University of Defense Technology(国防科技大学)