AI安全中的关键性概念
The Concept of Criticality in AI Safety
AI总结：
针对AI智能体价值对齐问题中全量人工监控效率低下的痛点，提出AI安全领域的关键性行动概念，让智能体仅就高风险关键行动申请人工许可，同时探讨了利用人工反馈优化智能体的路径。
摘要：
当AI智能体的行动与人类价值观不一致时，可能造成严重伤害。解决价值对齐问题的一种方法是安排人类操作员监控智能体的所有行动。尽管该方案能保障最高安全性，但效率极低，因为它要求人类操作员将全部注意力投入到智能体上。本文提出了一种效率高得多的解决方案，允许操作员在不忽视监控任务的同时参与其他活动。在我们的方法中，AI智能体仅针对关键性行动（即可能造成伤害的行动）向操作员请求许可。我们提出了AI安全语境下关键性行动的概念，并讨论了如何构建衡量行动关键性的模型。我们还探讨了如何利用操作员的反馈让智能体变得更智能。
英文摘要：
When AI agents don't align their actions with human values they may cause serious harm. One way to solve the value alignment problem is by including a human operator who monitors all of the agent's actions. Despite the fact, that this solution guarantees maximal safety, it is very inefficient, since it requires the human operator to dedicate all of his attention to the agent. In this paper, we propose a much more efficient solution that allows an operator to be engaged in other activities without neglecting his monitoring task. In our approach the AI agent requests permission from the operator only for critical actions, that is, potentially harmful actions. We introduce the concept of critical actions with respect to AI safety and discuss how to build a model that measures action criticality. We also discuss how the operator's feedback could be used to make the agent smarter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
