Krutrim LLM:面向超过十亿人的多语言基础模型
Krutrim LLM: Multilingual Foundational Model for over a Billion People
AI总结:
针对印度语言多样性和数据稀缺问题,提出2万亿词元的Krutrim多语言模型,采用最大印度语数据集,在印度语基准上超越或匹配SOTA,英语性能相当,并集成实时搜索,服务超10亿用户。
摘要:
印度是一个多元化的社会,在开发人工智能系统方面面临独特挑战,包括语言多样性、口头传统、数据可访问性和可扩展性。现有基础模型主要基于英语训练,限制了其对印度人口的有效性。尽管印度占全球人口的18%,但印度语言仅占Common Crawl语料库的1%,导致语言偏见。数千种区域语言、方言和代码混合由于训练数据稀疏,带来了额外的表征挑战。我们推出Krutrim LLM,这是一个为印度语言景观设计的2万亿词元多语言模型。它包含了已知最大的印度语数据集,缓解了数据稀缺问题,并确保各方言间的均衡性能。Krutrim在印度语基准测试上优于或匹配最先进的模型,同时保持有竞争力的英语性能。尽管训练浮点运算量显著较小,Krutrim LLM在16项任务中的10项上匹配或超过LLAMA-2等模型,平均得分为0.57对0.55。这证明了Krutrim在不同语言语境中灵活的多语言流畅性。Krutrim集成了实时搜索,以提高对话式AI应用中的事实准确性。这增强了对全球超过10亿用户的可访问性。通过解决数据失衡的有意设计选择,Krutrim LLM标志着在构建道德、全球代表性AI模型方面取得了有意义的进展。
英文摘要:
India is a diverse society with unique challenges in developing AI systems, including linguistic diversity, oral traditions, data accessibility, and scalability. Existing foundation models are primarily trained on English, limiting their effectiveness for India's population. Indic languages comprise only 1 percent of Common Crawl corpora despite India representing 18 percent of the global population, leading to linguistic biases. Thousands of regional languages, dialects, and code mixing create additional representation challenges due to sparse training data. We introduce Krutrim LLM, a 2 trillion token multilingual model designed for India's linguistic landscape. It incorporates the largest known Indic dataset, mitigating data scarcity and ensuring balanced performance across dialects. Krutrim outperforms or matches state-of-the-art models on Indic benchmarks while maintaining competitive English performance. Despite being significantly smaller in training flops, Krutrim LLM matches or exceeds models like LLAMA-2 on 10 out of 16 tasks, with an average score of 0.57 versus 0.55. This evidences Krutrim's flexible multilingual fluency across diverse linguistic contexts. Krutrim is integrated with real-time search to improve factual accuracy in conversational AI applications. This enhances accessibility for over 1 billion users worldwide. Through intentional design choices addressing data imbalances, Krutrim LLM signifies meaningful progress in building ethical, globally representative AI models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。