测量和评估用于诈骗检测的生成式人工智能模型的性能
Measuring and Evaluating the Performance of Generative AI Models for Scam Detection
浏览论文内容
中文总结 AI 辅助
研究大语言模型在无特定任务微调下检测诈骗的能力，通过策划发布独特基准数据集，评估九个不同模型，发现较大模型性能优，有效提示可提升小模型性能，且LLMs泛化能力强，还发布了数据集和评估框架。
中文摘要 AI 辅助
在线诈骗持续造成重大财务和个人伤害，基于大语言模型（LLMs）的检测系统已被集成到多种安全产品中。本文研究LLMs在无需特定任务微调的情况下能否有效检测各种场景下的诈骗。我们精心策划并发布了一个涵盖多种格式和主题的真实世界诈骗的独特基准数据集。评估了九个不同大小和架构的LLMs，研究它们在不同提示策略下的性能，并与基于BERT的微调分类器进行比较。结果表明，较大的LLMs通常比较小的表现更好，有效的提示可显著提高较小模型的性能，且LLMs在泛化到未见诈骗方面优于微调模型。我们还发布了数据集和评估框架以促进未来利用语言模型进行稳健诈骗检测的研究。
英文摘要
Online scams continue to cause substantial financial and personal harm. As a result, detection systems based on Large Language Models (LLMs) have been integrated into security products ranging from email gateways and browser extensions to fraud-monitoring dashboards. As this adoption accelerates, a common belief has taken hold: that these models are broadly suitable for scam detection. In this work, we investigate whether LLMs, with their strong capabilities in understanding intent, context, and reasoning, can effectively detect scams across diverse scenarios without task-specific fine-tuning. We curate and release a unique benchmark dataset of real-world scams spanning multiple formats and topics. We evaluate nine LLMs of varying sizes and architectures, examining their performance under different prompting strategies and comparing them to a fine-tuned BERT-based classifier. Our results show that while larger LLMs generally outperform smaller ones, effective prompting substantially boosts the performance of smaller models. Moreover, LLMs are better at generalizing to unseen scams compared to fine-tuned models, suggesting that pre-trained knowledge contributes meaningfully to scam detection. We release our dataset and evaluation framework to facilitate future research in robust scam detection using language models.
