AutoEvoEval：一个用于演化闭源式 LLM 评估数据的自动化框架
AutoEvoEval: An Automated Framework for Evolving Close-Ended LLM Evaluation Data
AI总结：
针对现有 LLM 评估基准静态且缺乏系统控制的问题，提出基于进化的框架 AutoEvoEval，引入 22 种原子操作和多轮组合，实验证明其能有效揭示模型在对抗扰动下的鲁棒性不足。
摘要：
大型语言模型（LLMs）在各种任务上表现出色，但现有的评估基准通常是静态的，不足以在现实场景中全面评估其鲁棒性和泛化能力。先前使用进化或对抗性数据增强的工作提高了评估多样性，但缺乏对扰动类型和多步复杂性的系统控制，限制了全面的鲁棒性分析。为了填补这些空白，我们提出了 AutoEvoEval，一个面向多选题问答等闭源式任务的基于进化的评估框架。AutoEvoEval 引入了 22 种可解释的原子进化操作，并支持多轮组合，从而能够可控地生成多样、具有挑战性且现实的测试样本。我们在广泛的开源和闭源 LLMs 上进行了大量实验，解答了四个研究问题。结果表明，原子操作导致平均准确率下降 7.283%，其中破坏结构或误导语义的编辑造成的下降幅度最大。对于相同的扰动，不同模型的敏感性差异显著，而结合多个进化步骤会使对抗效应放大高达 52.932%。这些发现表明，当前的基准可能高估了模型的真实泛化能力，并强调了进行进化感知鲁棒性评估的必要性。代码和资源可在 https://github.com/SYSUSELab/AutoEvoEval 获取。
英文摘要：
Large language models (LLMs) have shown remarkable performance on various tasks, but existing evaluation benchmarks are often static and insufficient to fully assess their robustness and generalization in realistic scenarios. Prior work using evolutionary or adversarial data augmentation has improved evaluation diversity but lacks systematic control over perturbation types and multi-step complexity, limiting comprehensive robustness analysis. To address these gaps, we propose AutoEvoEval, an evolution-based evaluation framework for close-ended tasks such as multi-choice question answering. AutoEvoEval introduces 22 interpretable atomic evolution operations and supports multi-round compositions, enabling controlled generation of diverse, challenging, and realistic test samples. We conduct extensive experiments addressing four research questions on a broad set of open- and closed-source LLMs. Our results show that atomic operations cause an average accuracy drop of 7.283\%, with structure-disrupting or misleading semantic edits causing the largest declines. Model sensitivities vary significantly for the same perturbation, and combining multiple evolution steps amplifies adversarial effects by up to 52.932\%. These findings suggest current benchmarks may overestimate true model generalization and emphasize the need for evolution-aware robustness evaluation. Code and resources are available at: https://github.com/SYSUSELab/AutoEvoEval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
