WebFAQ 2.0: 一个多语言问答数据集，包含挖掘的硬负样本用于密集检索
WebFAQ 2.0: A Multilingual QA Dataset with Mined Hard Negatives for Dense Retrieval
AI总结：
WebFAQ 2.0是一个多语言问答数据集，包含挖掘的硬负样本，用于训练密集检索器，支持多语言和跨语言信息检索研究。
摘要：
我们介绍了WebFAQ 2.0，一个新的WebFAQ数据集版本，包含19800万基于FAQ的自然问题-答案对，涵盖108种语言。与之前的版本相比，它显著扩展了多语言覆盖范围和双语对齐的QA对数量，达到超过1430万，使其成为最大的基于FAQ的资源。与原始发布不同，WebFAQ 2.0采用了一种新的数据收集策略，直接爬取和提取相关网络内容，从而产生了一个更加多样和多语言的数据集，通过页面标题和描述提供了更丰富的上下文。响应社区反馈，我们还发布了用于训练密集检索器的硬负样本数据集，包含20种语言的125万个查询。这些硬负样本使用两阶段检索流程挖掘，每个查询包含200个负样本的交叉编码分数。我们进一步展示了该资源如何使密集检索器的两种主要微调策略：基于多负样本排名损失的对比学习，以及基于边际MSE损失的知识蒸馏。WebFAQ 2.0不是一个静态资源，而是长期努力的一部分。自2025年底以来，通过开放网络索引定期发布结构化FAQ，使能够持续扩展和改进。我们发布数据集和训练脚本以促进进一步的多语言和跨语言信息检索研究。数据集本身及相关资源已公开在GitHub和HuggingFace上。
英文摘要：
We introduce WebFAQ 2.0, a new version of the WebFAQ dataset, containing 198 million FAQ-based natural question-answer pairs across 108 languages. Compared to the previous version, it significantly expands multilingual coverage and the number of bilingual aligned QA pairs to over 14.3M, making it the largest FAQ-based resource. Unlike the original release, WebFAQ 2.0 uses a novel data collection strategy that directly crawls and extracts relevant web content, resulting in a substantially more diverse and multilingual dataset with richer context through page titles and descriptions. In response to community feedback, we also release a hard negatives dataset for training dense retrievers, with 1.25M queries across 20 languages. These hard negatives were mined using a two-stage retrieval pipeline and include cross-encoder scores for 200 negatives per query. We further show how this resource enables two primary fine-tuning strategies for dense retrievers: Contrastive Learning with MultipleNegativesRanking loss, and Knowledge Distillation with MarginMSE loss. WebFAQ 2.0 is not a static resource but part of a long-term effort. Since late 2025, structured FAQs are being regularly released through the Open Web Index, enabling continuous expansion and refinement. We publish the datasets and training scripts to facilitate further research in multilingual and cross-lingual IR. The dataset itself and all related resources are publicly available on GitHub and HuggingFace.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
