arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.19844cs.CRcs.AIcs.SE
收藏

依赖大语言模型的应用需要系统级威胁监控

LLM-enabled Applications Require System-Level Threat Monitoring

Department of Computer Science, National University of Singapore, Singapore(新加坡国立大学计算机科学系);School of Computing and Information Systems, Singapore Management University, Singapore(新加坡管理学院计算机与信息系统学院)

Yedi Zhang, Haoyu Wang, Xianglin Yang, Jin Song Dong, Jun Sun

AI总结:

本文主张对依赖大语言模型的应用进行系统级威胁监控,以应对由此带来的安全挑战和确保可靠操作。

摘要:

依赖大语言模型的应用正在通过将大型语言模型作为复杂任务执行的核心推理组件,迅速重塑软件生态系统。然而,这种范式转变引入了根本性的可靠性挑战,并显著扩大了安全攻击面,由于LLM行为的非确定性、学习驱动性和难以验证的性质。鉴于这些新兴且不可避免的安全挑战,我们认为这些风险应被视为预期的操作条件而非异常事件,需要专门的事件响应视角。因此,可信部署的主要障碍不是进一步提高模型能力,而是建立系统级威胁监控机制,以在部署后检测和上下文化安全相关异常——这一方面在测试或基于护栏的防御之外 largely 未被深入探索。因此,本文主张对LLM应用中的安全威胁进行系统性和全面的监控,作为可靠操作的前提条件,以及专门事件响应框架的基础。

英文摘要:

LLM-enabled applications are rapidly reshaping the software ecosystem by using large language models as core reasoning components for complex task execution. This paradigm shift, however, introduces fundamentally new reliability challenges and significantly expands the security attack surface, due to the non-deterministic, learning-driven, and difficult-to-verify nature of LLM behavior. In light of these emerging and unavoidable safety challenges, we argue that such risks should be treated as expected operational conditions rather than exceptional events, necessitating a dedicated incident-response perspective. Consequently, the primary barrier to trustworthy deployment is not further improving model capability but establishing system-level threat monitoring mechanisms that can detect and contextualize security-relevant anomalies after deployment -- an aspect largely underexplored beyond testing or guardrail-based defenses. Accordingly, this position paper advocates systematic and comprehensive monitoring of security threats in LLM-enabled applications as a prerequisite for reliable operation and a foundation for dedicated incident-response frameworks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑