AI科学家的风险：优先保障安全而非自主性
Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy
AI总结：
本文分析AI科学家在自主科研中的潜在风险与脆弱性，提出人类监管、智能体对齐和环境反馈理解的三元框架，并呼吁完善模型、基准与法规以保障安全。
摘要：
由大型语言模型驱动的AI科学家已在自主开展实验和促进跨学科科学发现方面展现出巨大潜力。尽管其能力前景广阔，这些智能体也引入了新的脆弱性，需要为安全而审慎考量。然而，目前对这些脆弱性的全面探索仍然有限。本文从视角性论述出发，考察AI科学家中的脆弱性，揭示与其误用相关的潜在风险，并强调采取安全措施的必要性。我们首先概述AI科学家固有的潜在风险，综合考虑用户意图、具体科学领域及其对外部环境的潜在影响。随后，我们探讨这些脆弱性的深层原因，并对有限的现有工作进行范围综述。基于分析，我们提出一个三元框架，涉及人类监管、智能体对齐以及对环境反馈的理解（智能体监管），以缓解这些已识别的风险。此外，我们强调在保障AI科学家安全方面存在的局限与挑战，并倡导开发改进的模型、稳健的基准和全面的法规。
英文摘要：
AI scientists powered by large language models have demonstrated substantial promise in autonomously conducting experiments and facilitating scientific discoveries across various disciplines. While their capabilities are promising, these agents also introduce novel vulnerabilities that require careful consideration for safety. However, there has been limited comprehensive exploration of these vulnerabilities. This perspective examines vulnerabilities in AI scientists, shedding light on potential risks associated with their misuse, and emphasizing the need for safety measures. We begin by providing an overview of the potential risks inherent to AI scientists, taking into account user intent, the specific scientific domain, and their potential impact on the external environment. Then, we explore the underlying causes of these vulnerabilities and provide a scoping review of the limited existing works. Based on our analysis, we propose a triadic framework involving human regulation, agent alignment, and an understanding of environmental feedback (agent regulation) to mitigate these identified risks. Furthermore, we highlight the limitations and challenges associated with safeguarding AI scientists and advocate for the development of improved models, robust benchmarks, and comprehensive regulations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
