构建一个利用Rust中多进程设计的高效多语言非营利性伊斯兰领域信息检索系统
Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust
AI总结:
本文为伊斯兰领域构建多语言非营利信息检索系统,通过持续预训练和语言缩减获得轻量级模型,并利用Rust多进程实现低资源下的高效语义搜索。
摘要:
大型语言模型(LLMs)的广泛使用极大地改善了许多自然语言处理(NLP)应用,包括信息检索(IR)。然而,非商业利益驱动的领域在受益于人工智能解决方案方面往往滞后。其中一个领域是宗教和遗产语料库。与类似领域一样,伊斯兰文献具有重要的文化价值,并经常被学者和公众使用。浏览这些大量的文本具有挑战性,目前还没有一个统一的资源允许使用先进的AI工具轻松搜索这些数据。这项工作专注于为伊斯兰领域开发一个多语言非营利性信息检索系统。这个过程带来了一些主要挑战,例如在特定语言数据有限时准备多语言领域特定语料库,在资源受限的设备上部署模型,以及在有限预算下实现快速搜索。通过采用诸如持续预训练进行领域适应和语言缩减以减少模型大小等方法,准备了一个轻量级多语言检索模型,其性能优于在通用领域数据上预训练的更大模型。此外,评估所提出的利用Rust语言能力的架构表明,在低资源环境中实现高效语义搜索是可能的。
英文摘要:
The widespread use of large language models (LLMs) has dramatically improved many applications of Natural Language Processing (NLP), including Information Retrieval (IR). However, domains that are not driven by commercial interest often lag behind in benefiting from AI-powered solutions. One such area is religious and heritage corpora. Alongside similar domains, Islamic literature holds significant cultural value and is regularly utilized by scholars and the general public. Navigating this extensive amount of text is challenging, and there is currently no unified resource that allows for easy searching of this data using advanced AI tools. This work focuses on the development of a multilingual non-profit IR system for the Islamic domain. This process brings a few major challenges, such as preparing multilingual domain-specific corpora when data is limited in certain languages, deploying a model on resource-constrained devices, and enabling fast search on a limited budget. By employing methods like continued pre-training for domain adaptation and language reduction to decrease model size, a lightweight multilingual retrieval model was prepared, demonstrating superior performance compared to larger models pre-trained on general domain data. Furthermore, evaluating the proposed architecture that utilizes Rust Language capabilities shows the possibility of implementing efficient semantic search in a low-resource setting.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。