发表机构
Shanghai Artificial Intelligence Laboratory; The Chinese University of Hong Kong, Shenzhen; Tsinghua University（上海人工智能实验室; 香港中文大学（深圳）; 清华大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对由LLM驱动的智能体AI系统，基于物理、社会、自我指涉认知的三级框架分析其引发的风险，提出策略以增强系统可控性，保障长期安全发展。
AI 中文摘要
由大语言模型（LLM）驱动的前沿智能体系统展现出类人的认知模式。随着这些系统深度融入不同领域，其认知参与引发了人类社会尚未充分研究的关键关切。为填补这一空白，我们遵循认知范围定义的三级框架（从物理认知到社会认知，再到自我指涉认知），系统分析了认知能力扩展引发的风险。针对每个认知层级，我们研究了它们对人类能动性、自主性和控制能力的潜在风险，最终提出缓解这些风险、增强智能体AI系统可控性的策略，以确保其长期安全发展。
英文摘要
Frontier agentic systems powered by large language models (LLMs) exhibit human-like patterns of cognition. As these systems become deeply integrated across different domains, their cognitive engagement raises critical concerns for human society that remain insufficiently studied. To address this gap, we systematically analyze risks induced by expanding cognitive capabilities, following a three-level framework defined by their cognitive scope, from physical cognition to social cognition, and finally to self-referential cognition. We study their potential risks to human agency, autonomy, and control capability, corresponding to each cognitive level. We finally propose strategies to mitigate these risks and enhance the controllability of agentic AI systems, ensuring their long-term safe development.
CommentsThis paper has been accepted by IEEE Intelligent Systems, which can be accessed at https://doi.ieeecomputersociety.org/10.1109/MIS.2026.3721766. The DOI is 10.1109/MIS.2026.3721766
