调查多语言先前已事实核查声明检测中的语言与检索偏差
Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection
AI总结：
本文面向 PFCD 任务，基于 AMC-16K 评估六个开源多语言 LLM 在20种语言中的表现，并分析多语言嵌入检索的频率偏差，揭示语言与检索不公平现象及改进方向。
摘要：
多语言大型语言模型（LLM）为跨语言事实核查提供了强大能力。然而，这些模型往往表现出语言偏差，在英语等高资源语言上的表现显著优于低资源语言。我们还提出并考察了一个新概念——检索偏差，即信息检索系统倾向于偏好某些信息而非其他信息，从而使检索过程发生偏斜。本文在先前已事实核查声明检测（PFCD）背景下研究语言偏差和检索偏差。我们利用 AMC-16K 数据集，采用完全多语言提示策略，在20种语言上评估了六个开源多语言 LLM。通过将任务提示翻译成每种语言，我们揭示了单语言和跨语言性能方面的差异，并基于模型系列、规模和提示策略识别出关键趋势。研究结果突显了 LLM 行为中持续存在的偏差，并为提升多语言事实核查的公平性提供了建议。为调查检索偏差，我们采用多语言嵌入模型，并研究被检索声明的频率。分析表明，某些声明在不同帖子中被不成比例地检索，导致热门声明的检索性能被抬高，而较不常见的声明则代表性不足。
英文摘要：
Multilingual Large Language Models (LLMs) offer powerful capabilities for cross-lingual fact-checking. However, these models often exhibit language bias, performing disproportionately better on high-resource languages such as English than on low-resource counterparts. We also present and inspect a novel concept - retrieval bias, when information retrieval systems tend to favor certain information over others, leaving the retrieval process skewed. In this paper, we study language and retrieval bias in the context of Previously Fact-Checked Claim Detection (PFCD). We evaluate six open-source multilingual LLMs across 20 languages using a fully multilingual prompting strategy, leveraging the AMC-16K dataset. By translating task prompts into each language, we uncover disparities in monolingual and cross-lingual performance and identify key trends based on model family, size, and prompting strategy. Our findings highlight persistent bias in LLM behavior and offer recommendations for improving equity in multilingual fact-checking. To investigate retrieval bias, we employed multilingual embedding models and look into the frequency of retrieved claims. Our analysis reveals that certain claims are retrieved disproportionately across different posts, leading to inflated retrieval performance for popular claims while under-representing less common ones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
