当规模遇上多样性：在细粒度多语言声明验证上评估语言模型
When Scale Meets Diversity: Evaluating Language Models on Fine-Grained Multilingual Claim Verification
AI总结：
本文在X-Fact上评估五类模型，发现270M参数的XLM-R经微调后以57.7% macro-F1大幅超越7B-12B LLM，并揭示其证据利用困难与类别偏见。
摘要：
多语言错误信息的快速传播需要稳健的自动化事实验证系统，以处理跨多种语言的细粒度真实性评估。尽管大型语言模型已在许多NLP任务中展现出卓越能力，但它们在具有细致分类体系的多语言声明验证中的有效性仍研究不足。我们在涵盖25种语言、包含七个不同真实性类别的X-Fact数据集上，对五个最先进语言模型进行了综合评估。实验使用提示和微调两种方法，比较了小型语言模型（基于编码器的XLM-R和mT5）与近期仅解码器LLM（Llama 3.1、Qwen 2.5、Mistral Nemo）。令人惊讶的是，我们发现XLM-R（2.7亿参数）显著优于所有受测LLM（70亿至120亿参数），取得57.7%的macro-F1，而最佳LLM性能仅为16.9%。这比此前最先进水平（41.9%）提升15.8%，为多语言事实验证建立了新的性能基准。分析揭示了LLM行为中的问题模式，包括在利用证据方面的系统性困难，以及在不平衡数据设置中对高频类别的明显偏见。这些发现表明，对于细粒度多语言事实验证，较小的专用模型可能比通用大型模型更有效，这对事实核查系统的实际部署具有重要意义。
英文摘要：
The rapid spread of multilingual misinformation requires robust automated fact verification systems capable of handling fine-grained veracity assessments across diverse languages. While large language models have shown remarkable capabilities across many NLP tasks, their effectiveness for multilingual claim verification with nuanced classification schemes remains understudied. We conduct a comprehensive evaluation of five state-of-the-art language models on the X-Fact dataset, which spans 25 languages with seven distinct veracity categories. Our experiments compare small language models (encoder-based XLM-R and mT5) with recent decoder-only LLMs (Llama 3.1, Qwen 2.5, Mistral Nemo) using both prompting and fine-tuning approaches. Surprisingly, we find that XLM-R (270M parameters) substantially outperforms all tested LLMs (7-12B parameters), achieving 57.7% macro-F1 compared to the best LLM performance of 16.9%. This represents a 15.8% improvement over the previous state-of-the-art (41.9%), establishing new performance benchmarks for multilingual fact verification. Our analysis reveals problematic patterns in LLM behavior, including systematic difficulties in leveraging evidence and pronounced biases toward frequent categories in imbalanced data settings. These findings suggest that for fine-grained multilingual fact verification, smaller specialized models may be more effective than general-purpose large models, with important implications for practical deployment of fact-checking systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
