审视AI陪伴应用生态系统的风险
Examining Risks Through a Characterization of the AI Companion Application Ecosystem: A Stratified Sample from the Apple App Store and Google Play Store
AI总结：
本文通过分析489个AI陪伴应用，揭示了用户面临的风险及恶意用户利用应用功能带来的威胁，重点关注敏感数据收集、拟人化、互动机制及合成非自愿亲密影像等问题。
摘要：
尽管允许用户通过自然语言交互的计算机系统（即聊天机器人）已存在多年，但近年来随着大语言模型的发展，各种声称提供AI陪伴的应用（如Character AI、Replika）迅速增多。我们的工作提出了一个涵盖两个不同风险类别的威胁模型：由AI陪伴应用对用户造成的危害，以及恶意用户利用应用功能所带来的危害。为了进一步理解这个应用生态系统，我们从App Store和Play Store中识别出489个声称提供AI陪伴的应用。然后，我们系统地对30个应用进行了分层抽样，并根据我们的威胁模型进行了分析。通过我们的分析，我们分类了更广泛的生态系统趋势，为理解威胁提供了背景，并识别了与敏感数据收集和共享、拟人化、互动机制、性互动和媒体、以及 likeness 的摄入和重建相关的具体威胁，包括生成合成非自愿亲密影像的潜在可能性。本研究为AI陪伴应用生态系统的安全视角提供了基础，并为未来该领域内及之外的研究、政策和技术发展提供了信息。
英文摘要：
While computer systems that allow users to interact through conversational natural language (i.e., chatbots) have existed for many years, various types of applications offering AI companionship (e.g., Character AI, Replika) have proliferated in recent years due to advancements in large language models. To better understand this application ecosystem, we identified 489 unique apps from the Apple App Store and Google Play Store that advertised AI companionship with social or relational capabilities (e.g., an AI romantic partner). We then systematically conducted and analyzed walkthroughs of a stratified sample of 30 apps, focusing on two distinct risk categories: potential harms posed to users by AI companion apps, and potential harms enabled by malicious users exploiting app features. Through our analysis, we categorize broader ecosystem trends that provide context for understanding risks and identify specific risks related to sensitive data collection and sharing, anthropomorphism, engagement mechanisms, sexual content, as well as the ingestion and reconstruction of likeness, including the potential for generating synthetic nonconsensual intimate imagery (synthetic NCII). We conclude with a discussion of paths for different key stakeholders to mitigate the identified risks. Content warning: This paper includes descriptions of applications that can be used to create synthetic nonconsensual representations, including intimate imagery, as well as discussion of suicidal ideation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
