We are currently in an era of fierce competition among various large language models (LLMs) continuously pushing the boundaries of benchmark performance. However, genuinely assessing the capabilities of these LLMs has become a challenging and critical issue due to potential data contamination, and it wastes dozens of time and effort for researchers and engineers to download and try those contaminated models. To save our precious time, we propose a novel and useful method, Clean-Eval, which mitigates the issue of data contamination and evaluates the LLMs in a cleaner manner. Clean-Eval employs an LLM to paraphrase and back-translate the contaminated data into a candidate set, generating expressions with the same meaning but in different surface forms. A semantic detector is then used to filter the generated low-quality samples to narrow down this candidate set. The best candidate is finally selected from this set based on the BLEURT score. According to human assessment, this best candidate is semantically similar to the original contamination data but expressed differently. All candidates can form a new benchmark to evaluate the model. Our experiments illustrate that Clean-Eval substantially restores the actual evaluation results on contaminated LLMs under both few-shot learning and fine-tuning scenarios.
Beyond Clean Text: Evaluating Encoder and Decoder Robustness for Bangla Event Detection in Noisy Text
Jahangirnagar University(贾汉吉尔纳加尔大学);Rajshahi University of Engineering & Technology(拉杰沙希工程技术大学);Bangladesh University of Engineering and Technology(孟加拉国工程技术大学)
Establishing Trustworthy LLM Evaluation via Shortcut Neuron Analysis
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(认知与决策智能复杂系统重点实验室,自动化研究所,中国科学院,北京,中国);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学);Tsinghua University(清华大学)
When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation
Saarland University(萨尔兰大学);Queen’s University Belfast(贝尔法斯特女王大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);University of Edinburgh(爱丁堡大学)
Data Contamination in Neural Hieroglyphic Translation: A Reproducibility Study
Computer Science and Engineering, Alamein International University (AIU)(阿尔梅因国际大学(AIU)计算机科学与工程系);HiTZ Center, University of the Basque Country(巴斯克大学HiTZ中心);Faculty of Computers and Data Science, Alexandria University(亚历山大大学计算机与数据科学学院)