超越静态基准：基于人设的模拟合成有害内容以实现稳健评估
Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation
AI总结：
本文提出基于人设的模拟合成有害内容框架，通过整合人口统计信息与主题兴趣，生成多样化且情境相关的有害交互，验证了其在有害性、挑战性和多样性方面的有效性，为评估检测系统提供更鲁棒的测试工具。
摘要：
静态基准在有害内容检测中面临可扩展性和多样性限制，并可能受网络级预训练语料污染。为解决这些问题，我们提出一个框架，利用基于人设的大语言模型（LLM）代理合成有害内容。我们的方法通过整合人口统计身份和主题兴趣与情境有害策略，构建二维用户人设，实现多样化且情境相关的有害交互模拟。我们从有害性、挑战级别和多样性三个维度评估该框架。人类和基于LLM的评估均证实，我们的框架实现了高有害生成成功率。在多个检测系统上的实验表明，我们的合成场景比现有基准更具挑战性。此外，多方面的分析证实，我们的方法在语言和主题多样性方面与人工整理的数据集相当，确立了该框架作为稳健压力测试有害内容检测系统的有效工具。
英文摘要：
Static benchmarks for harmful content detection face limitations in scalability and diversity, and may also be affected by contamination from web-scale pre-training corpora. To address these issues, we propose a framework for synthesizing harmful content, leveraging persona-guided large language model (LLM) agents. Our approach constructs two-dimensional user personas by integrating demographic identities and topical interests with situational harmful strategies, enabling the simulation of diverse and contextually grounded harmful interactions. We evaluate the framework along three dimensions: harmfulness, challenge level, and diversity. Both human and LLM-based evaluations confirm that our framework achieves a high harmful generation success rate. Experiments across multiple detection systems reveal that our synthetic scenarios are more challenging to detect than those in existing benchmarks. Furthermore, a multi-faceted analysis confirms that our approach achieves linguistic and topical diversity comparable to human-curated datasets, establishing our framework as an effective tool for robust stress-testing of harmful content detection systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
