arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.00323cs.CLcs.CY
收藏

大语言模型在滥用检测流水线中的应用

Large Language Models in the Abuse Detection Pipeline

Google LLC(谷歌有限责任公司)

Suraj Kath, Sanket Badhe, Preet Shah, Ashwin Sampathkumar, Shivani Gupta

AI总结:

本文探讨大语言模型在滥用检测生命周期中的应用,分析其在标签生成、检测、审核及审计等阶段的作用,并讨论其面临的挑战与未来研究方向。

摘要:

在线滥用行为日益复杂,涵盖有毒语言、骚扰、操纵和欺诈行为。传统机器学习方法依赖静态分类器和人工标注,难以应对不断演变的威胁模式和复杂的政策要求。大语言模型引入了上下文推理、政策解释、解释生成和跨模态理解等能力,使其能够支持现代安全系统的多个阶段。本文从生命周期角度分析大语言模型如何整合到滥用检测生命周期(ADL)中,涵盖四个阶段:(I)标签与特征生成,(II)检测,(III)审核与申诉,(IV)审计与治理。针对每个阶段,本文综合了新兴研究和行业实践,强调生产部署的架构考虑,并检验大语言模型驱动方法的优缺点。最后,本文概述了关键挑战,包括延迟、成本效率、确定性、对抗鲁棒性和公平性,并讨论了未来研究方向,以将大语言模型作为可靠、可问责的大型滥用检测和治理系统组件加以应用。

英文摘要:

Online abuse has grown increasingly complex, spanning toxic language, harassment, manipulation, and fraudulent behavior. Traditional machine-learning approaches dependent on static classifiers and labor-intensive labeling struggle to keep pace with evolving threat patterns and nuanced policy requirements. Large Language Models introduce new capabilities for contextual reasoning, policy interpretation, explanation generation, and cross-modal understanding, enabling them to support multiple stages of modern safety systems. This survey provides a lifecycle-oriented analysis of how LLMs are being integrated into the Abuse Detection Lifecycle (ADL), which we define across four stages: (I) Label \& Feature Generation, (II) Detection, (III) Review \& Appeals, and (IV) Auditing \& Governance. For each stage, we synthesize emerging research and industry practices, highlight architectural considerations for production deployment, and examine the strengths and limitations of LLM-driven approaches. We conclude by outlining key challenges including latency, cost-efficiency, determinism, adversarial robustness, and fairness and discuss future research directions needed to operationalize LLMs as reliable, accountable components of large-scale abuse-detection and governance systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑