发表机构
Microsoft Research; Duke University; Abridge AI Inc.; Microsoft（微软研究院; 杜克大学; Abridge人工智能公司; 微软）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文通过访谈19名儿童安全相关从业者，分析现有AI聊天机器人儿童安全评估的局限，提出需将从业者视角纳入安全评估工作的建议。
AI 中文摘要
青少年越来越多地向AI聊天机器人寻求社会和情感支持，这引发了人们对这些系统如何回应的担忧，尤其是在高风险情境下。然而，现有的针对AI的儿童安全评估缺乏对青少年实际经历的真实伤害的依据，依赖于关于什么是适当输出（例如拒绝）的未经验证的假设，并且通常仅聚焦于检测对抗性提示或输出中的表面伤害。因此，这些评估可能无法检测到在实践中对青少年造成伤害的回应。为了更好地理解当前评估实践的局限性，我们对19名直接与处于弱势情境的青少年合作的从业者进行了访谈，其中包括社会工作者、治疗师和心理学家，要求他们反思聊天机器人对青少年常见风险情境的回应，这些情境已在先前的实证研究中确立。从业者确定了聊天机器人可能造成伤害的行为，以及那些能够在困难时刻为青少年提供有意义支持的行为，讨论了聊天机器人在这些互动中应扮演（和不应扮演）的角色，并提供了改进聊天机器人回应的具体建议。基于这些发现，我们为AI儿童安全评估和基础设施提供了建议，并强调需要将从业者的观点纳入安全工作中。
英文摘要
Youth increasingly turn to AI chatbots for social and emotional support, raising concerns about how these systems respond, especially in high-stakes situations. However, existing child safety evaluations of AI lack grounding in real-world harms that youth experience, rely on unvalidated assumptions about what counts as an appropriate output (e.g., refusal), and typically focus on detecting adversarial prompts or surface-level harms in outputs only. Thus, these evaluations can fail to detect responses that pose harm to youth in practice. To better understand the limitations of current evaluation practices, we conducted interviews with 19 practitioners working directly with youth in vulnerable situations, including social workers, therapists, and psychologists, asking them to reflect on chatbots' responses to risky situations commonly faced by youth, as established in prior empirical work. Practitioners identified chatbot behaviors likely to cause harm as well as those that could meaningfully support youth in difficult moments, discussed the role that chatbots should (and should not) play in these interactions, and offered concrete recommendations for improving chatbot responses. Based on these findings, we provide recommendations for AI child safety evaluation and infrastructure, and highlight the need for incorporating practitioners' perspectives into safety work.
CommentsNinth AAAI/ACM Conference on AI, Ethics, and Society (AIES 2026)
