DeepRAG：从零构建用于检索增强生成的定制印地语嵌入模型
DeepRAG: Building a Custom Hindi Embedding Model for Retrieval Augmented Generation from Scratch
AI总结：
本文提出DeepRAG，从零构建印地语专用嵌入模型，通过270万样本、定制分词器、印地语特定注意力机制和对比学习，使检索精度比多语言模型提升23%，并集成LangChain构建完整印地语RAG系统。
摘要：
本文介绍了我们在DeepRAG上的工作，这是一个我们专门为RAG系统中的印地语构建的专用嵌入模型。尽管大语言模型（LLMs）在文本生成方面已经表现得非常出色，但它们在检索任务中的性能仍然高度依赖于高质量的嵌入——而这一点对于印地语来说一直有所欠缺，尽管印地语是世界上使用人数最多的语言之一。我们通过从零开始创建嵌入来解决这一问题，而不是仅仅微调现有模型。我们的流程包括收集多样化的印地语文本（超过270万个样本）、训练一个真正理解印地语形态学的定制SentencePiece分词器、设计具有印地语特定注意力机制的Transformer架构，以及使用对比学习进行优化。结果确实比我预期的要好——与大家一直在使用的多语言模型相比，我们的检索精度提高了23%。本文详细介绍了我们的方法，我认为这可以帮助其他从事低资源语言研究的人，因为一刀切的多语言模型在这些语言上表现不佳。我们还将我们的嵌入与LangChain集成，构建了完整的印地语RAG系统，这对实践者可能有用。虽然还有很多内容需要探索，但我相信这项工作填补了印地语NLP的一个关键空白，并证明了语言特定方法的重要性。
英文摘要：
In this paper, I present our work on DeepRAG, a specialized embedding model we built specifically for Hindi language in RAG systems. While LLMs have gotten really good at generating text, their performance in retrieval tasks still depends heavily on having quality embeddings - something that's been lacking for Hindi despite being one of the world's most spoken languages. We tackled this by creating embeddings from the ground up rather than just fine-tuning existing models. Our process involved collecting diverse Hindi texts (over 2.7M samples), training a custom SentencePiece tokenizer that actually understands Hindi morphology, designing transformer architecture with Hindi-specific attention mechanisms, and optimizing with contrastive learning. Results were honestly better than I expected - we saw a 23% improvement in retrieval precision compared to the multilingual models everyone's been using. The paper details our methodology, which I think could help others working with low-resource languages where the one-size-fits-all multilingual models fall short. We've also integrated our embeddings with LangChain to build complete Hindi RAG systems, which might be useful for practitioners. While there's still tons more to explore, I believe this work addresses a critical gap for Hindi NLP and demonstrates why language-specific approaches matter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
