arXiv 2402.131092024-06-04cs.CLcs.AI

CIF-Bench：用于评估大型语言模型泛化能力的中文指令遵循基准

CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models

HKUST（香港科技大学）；Queen Mary University of London（伦敦大学玛丽皇后学院）；Ohio State University（俄亥俄州立大学）；AIWaves Inc.（AIWaves公司）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, Chinese Academy of Sciences（中国科学院人工智能学院）；Peking University（北京大学）；Beijing Foreign Studies University（北京外国语大学）；University of Manchester（曼彻斯特大学）；University of Waterloo（滑铁卢大学）；National University of Singapore（新加坡国立大学）；Zhejiang University（浙江大学）；Beihang University（北京航空航天大学）

Yizhi LI, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Zekun Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, Wangchunshu Zhou, Yiming Liang, Lei Zhang, Lei Ma,… 展开作者

Yizhi LI, Ge Zhang, Xingwei Qu, Jiali Li, Zhaoqun Li, Zekun Wang, Hao Li, Ruibin Yuan, Yinghao Ma, Kai Zhang, Wangchunshu Zhou, Yiming Liang, Lei Zhang, Lei Ma, Jiajun Zhang, Zuowen Li, Stephen W. Huang, Chenghua Lin, Jie Fu

备注：: Camera-ready version for ACL 2024. Project page at https://yizhilll.github.io/CIF-Bench/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CIF-Bench中文指令遵循基准，包含150个任务和1.5万个输入输出对，通过零样本评估28个LLM，揭示其在中文等低资源语言和复杂任务上的泛化局限，为未来模型发展设定新标准。

摘要：

大型语言模型（LLMs）的进步增强了其通过指令遵循在广泛未见自然语言处理（NLP）任务上的泛化能力。然而，它们在中文等低资源语言中的有效性通常会降低，加之数据泄露导致的评估偏差，引发了对其向新语言领域真实泛化能力的质疑。为此，我们引入了中文指令遵循基准（CIF-Bench），旨在评估LLMs对中文的零样本泛化能力。CIF-Bench包含150个任务和15,000个输入-输出对，由母语者开发，以测试跨越20个类别的复杂推理和中国文化细微差别。为缓解数据污染，我们仅公开发布一半数据集，另一半保持私有，并引入多样化指令以最小化分数方差，总计包含45,000个数据实例。我们对选定的28个LLMs的评估揭示了显著的性能差距，表现最好的模型得分仅为52.9%，凸显了LLMs在较不熟悉的语言和任务语境中的局限性。这项工作不仅揭示了当前LLMs在处理中文任务方面的局限性，还为未来的LLM泛化研究设定了新标准，推动开发更具适应性、文化感知力和语言多样性的模型。

英文摘要：

The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-following. Yet, their effectiveness often diminishes in low-resource languages like Chinese, exacerbated by biased evaluations from data leakage, casting doubt on their true generalizability to new linguistic territories. In response, we introduce the Chinese Instruction-Following Benchmark (CIF-Bench), designed to evaluate the zero-shot generalizability of LLMs to the Chinese language. CIF-Bench comprises 150 tasks and 15,000 input-output pairs, developed by native speakers to test complex reasoning and Chinese cultural nuances across 20 categories. To mitigate data contamination, we release only half of the dataset publicly, with the remainder kept private, and introduce diversified instructions to minimize score variance, totaling 45,000 data instances. Our evaluation of 28 selected LLMs reveals a noticeable performance gap, with the best model scoring only 52.9%, highlighting the limitations of LLMs in less familiar language and task contexts. This work not only uncovers the current limitations of LLMs in handling Chinese language tasks but also sets a new standard for future LLM generalizability research, pushing towards the development of more adaptable, culturally informed, and linguistically diverse models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。