发表机构
DLR Institute for AI Safety and Security（德国航空航天中心人工智能安全与安保研究所）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过范围综述和VR实验，探讨RLHF在人机协作中的应用，发现用户发起的反馈比系统发起的反馈更能有效捕捉感知安全性，强调反馈时机对AI安全的重要性。
AI 中文摘要
人机协作（HRC）可促进工业4.0中的大规模定制，而基于人类反馈的强化学习（RLHF）是开发安全的人工智能机器人的一种有前景的方法。在人工智能开发过程中的安全性、人类反馈质量以及人机双向适应方面仍存在实际挑战。我们对HRC系统中的RLHF进行了范围综述，梳理了应对这些挑战的方法。遵循PRISMA指南，我们筛选了199条记录，并纳入了20篇同行评审的出版物（2020-2025年），涵盖多个HRC领域。据我们所知，这是首个聚焦于RLHF双向闭环设计的综述。我们的综述发现，多种反馈模态能够以不同反馈格式收集数据。收集的数据可在人工智能训练的不同阶段进行整合，从而形成多步骤的开发过程。试点实验通常用于基于人类和机器人指标评估HRC系统。为了实证检验综述中识别的一个关键空白，我们进行了一项受试者间VR实验，比较了系统发起和用户发起的反馈对机器人近身行为（用于安全导航）的影响。使用贝叶斯模型，我们分析了收集的反馈与安全指标之间的关系：心理安全（实验后问卷）和物理安全（逆碰撞时间）。结果表明，用户发起的反馈比系统发起的反馈更能捕捉感知安全性，表明反馈时机直接影响反馈质量。我们的综述和实验发现表明，RLHF依赖于适当的反馈方法来确保HRC中的人工智能安全，未来的RLHF研究应优先考虑真实的HRC实验，评估反馈收集方法对相关人类和机器人指标的影响。
英文摘要
Human-Robot Collaboration (HRC) can facilitate mass customisation in Industry 4.0, with Reinforcement Learning from Human Feedback (RLHF) representing a promising approach for developing safe AI-based robots. Practical challenges remain regarding safety during AI development, human feedback quality, and bidirectional human-robot adaptation. We conducted a scoping review of RLHF in HRC systems, mapping methods that address these challenges. Following PRISMA guidelines, we screened 199 records and included 20 peer-reviewed publications (2020-2025) spanning multiple HRC domains. To our knowledge, this is the first review focused on the bidirectional, closed-loop design of RLHF. Our review found multiple feedback modalities enabling data collection in various feedback formats. Collected data can be integrated at different stages of AI training, resulting in a multi-step development process. Pilot experiments are commonly used to evaluate HRC systems based on both human and robot metrics. To empirically test a key gap identified in the review, we conducted a between-subjects VR experiment comparing system- and user-initiated feedback on robot proxemic behaviour for safe navigation. Using Bayesian models, we analysed the relation between the collected feedback and safety metrics: psychological safety (post-experiment questionnaire) and physical safety (inverse time-to-collision). Results show that user-initiated feedback captures perceived safety better than system-initiated feedback, indicating that feedback timing directly affects feedback quality. Our review and experiment findings show that RLHF relies on appropriate feedback methods to ensure AI safety in HRC, and future RLHF research should prioritise realistic HRC experiments evaluating the effects of feedback collection methods on relevant human and robot metrics.
