arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.09154cs.CL
收藏

CLEAN-EVAL:在受污染的大型语言模型上的干净评估

CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models

Shanghai Jiao Tong University(上海交通大学);FaceMind Corporation(FaceMind公司);Tsinghua University(清华大学);University of Edinburgh(爱丁堡大学)

Wenhong Zhu, Hongkun Hao, Zhiwei He, Yunze Song, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, Hongyuan Lu

AI总结:

针对大型语言模型的数据污染评估难题,提出Clean-Eval方法,通过改写、回译和BLEURT评分筛选生成新基准,在少样本和微调场景下有效恢复了受污染模型的实际评估结果。

摘要:

我们目前正处于各种大型语言模型(LLMs)激烈竞争的时代,这些模型不断突破基准性能的边界。然而,由于潜在的数据污染,真正评估这些LLMs的能力已成为一个具有挑战性且关键的问题,并且浪费了研究人员和工程师大量时间和精力去下载和尝试那些受污染的模型。为了节省我们宝贵的时间,我们提出了一种新颖且有用的方法Clean-Eval,它缓解了数据污染问题,并以更干净的方式评估LLMs。Clean-Eval利用一个LLM对受污染数据进行改写和回译,生成一个候选集,产生具有相同含义但表面形式不同的表达。然后使用一个语义检测器来过滤生成的低质量样本,以缩小这个候选集。最终基于BLEURT分数从该集合中选择最佳候选。根据人工评估,这个最佳候选在语义上与原始污染数据相似,但表达方式不同。所有候选可以组成一个新的基准来评估模型。我们的实验表明,Clean-Eval在少样本学习和微调场景下均大幅恢复了受污染LLMs的实际评估结果。

英文摘要:

We are currently in an era of fierce competition among various large language models (LLMs) continuously pushing the boundaries of benchmark performance. However, genuinely assessing the capabilities of these LLMs has become a challenging and critical issue due to potential data contamination, and it wastes dozens of time and effort for researchers and engineers to download and try those contaminated models. To save our precious time, we propose a novel and useful method, Clean-Eval, which mitigates the issue of data contamination and evaluates the LLMs in a cleaner manner. Clean-Eval employs an LLM to paraphrase and back-translate the contaminated data into a candidate set, generating expressions with the same meaning but in different surface forms. A semantic detector is then used to filter the generated low-quality samples to narrow down this candidate set. The best candidate is finally selected from this set based on the BLEURT score. According to human assessment, this best candidate is semantically similar to the original contamination data but expressed differently. All candidates can form a new benchmark to evaluate the model. Our experiments illustrate that Clean-Eval substantially restores the actual evaluation results on contaminated LLMs under both few-shot learning and fine-tuning scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑