GAIA Search：面向 NLP 训练数据探索的 Hugging Face 与 Pyserini 互操作性
GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration
AI总结：
本文提出将 Pyserini 与 Hugging Face 生态集成，并构建 GAIA Search，以支持 NLP 研究人员在训练前便捷检索和定性分析四个大规模文本数据集。
摘要：
注意到迫切需要提供工具，以便对现代 NLP 中的大规模文本语料库进行快速且用户友好的定性分析，我们建议求助于信息检索（IR）领域中成熟且经过充分检验的方法——这一研究领域在处理 TB 级文档集合方面具有悠久历史。我们讨论了如何将 Pyserini——一个被广泛使用、用于可复现 IR 研究的工具包——与由开源 AI 库和制品组成的 Hugging Face 生态系统集成。我们利用两个平台的现有功能，同时提出进一步促进其集成的新特性。我们的目标是为 NLP 研究人员提供工具，使其能够轻松、敏捷地开发满足自身数据分析需求的基于检索的工具。我们提供了一个基于 Jupyter Notebook 的核心互操作功能演示，可在 GitHub 的 https://github.com/huggingface/gaia 获取。随后，我们展示了如何将所提出的想法付诸实施，以创建一个用于 NLP 定性数据分析的强大工具。我们介绍了 GAIA Search，这是一个遵循此前阐述原则构建的搜索引擎，可访问四个流行的大规模文本集合。GAIA 具有双重用途：既展示我们所讨论方法的潜力，也可作为独立的定性分析工具，供希望在训练前理解数据集的 NLP 研究人员使用。GAIA 已托管在 Hugging Face Spaces 上：https://huggingface.co/spaces/spacerini/gaia。
英文摘要：
Noticing the urgent need to provide tools for fast and user-friendly qualitative analysis of large-scale textual corpora of the modern NLP, we propose to turn to the mature and well-tested methods from the domain of Information Retrieval (IR) - a research field with a long history of tackling TB-scale document collections. We discuss how Pyserini - a widely used toolkit for reproducible IR research can be integrated with the Hugging Face ecosystem of open-source AI libraries and artifacts. We leverage the existing functionalities of both platforms while proposing novel features further facilitating their integration. Our goal is to give NLP researchers tools that will allow them to develop retrieval-based instrumentation for their data analytics needs with ease and agility. We include a Jupyter Notebook-based walk through the core interoperability features, available on GitHub at https://github.com/huggingface/gaia. We then demonstrate how the ideas we present can be operationalized to create a powerful tool for qualitative data analysis in NLP. We present GAIA Search - a search engine built following previously laid out principles, giving access to four popular large-scale text collections. GAIA serves a dual purpose of illustrating the potential of methodologies we discuss but also as a standalone qualitative analysis tool that can be leveraged by NLP researchers aiming to understand datasets prior to using them in training. GAIA is hosted live on Hugging Face Spaces - https://huggingface.co/spaces/spacerini/gaia.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
