arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.22290cs.CLcs.IR
收藏

少即是多:利用小规模噪声合成数据适应低资源语言的文本嵌入

Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data

Metric AI Lab(Metric AI实验室)

Zaruhi Navasardyan, Spartak Bughdaryan, Bagrat Minasyan, Hrant Davtyan

AI总结:

本文挑战了大规模数据对语义对齐的必要性假设,通过小规模噪声合成数据提升低资源语言文本嵌入性能,实验显示在仅1万对数据下即可获得显著改进,验证了噪声对语义对齐的鲁棒性。

摘要:

低资源语言(LRLs)通常缺乏高质量大规模数据集,阻碍了文本嵌入模型在检索增强生成(RAG)和语义搜索中的应用。本文挑战了语义对齐需要大规模数据或纯净人工翻译的假设。以亚美尼亚语(一种具有独特书写的LRL)为例,引入了一种低成本的适应策略,利用由英文Reddit标题-正文对翻译生成的小规模噪声合成数据。我们建立了包含现有数据集、翻译数据和人工整理数据集的全面评估基准。实验揭示了“少即是多”现象:在仅1万对噪声合成数据上微调多语言编码器(mE5)可使基准测试平均提升11-12%,检索性能提升20%以上,与训练100万例的数据模型性能相当。进一步表明,增加数据规模、提升翻译质量或多样化数据领域均无法超越这一最小基线。我们验证了这些发现在另一种具有独特书写的LRL上的泛化性。结果表明,LRL的语义对齐早期饱和且对噪声高度鲁棒,使资源受限社区能够实现高性能嵌入创建。我们发布了模型、数据和基准测试,以促进进一步研究。

英文摘要:

Low-resource languages (LRLs) often lack high-quality, large-scale datasets for training effective text embedding models, hindering their application in tasks like retrieval-augmented generation (RAG) and semantic search. In this work, we challenge the prevailing assumption that effective semantic alignment requires massive datasets or pristine, human-verified translations. Focusing on Armenian (an LRL with a unique script), we introduce a cost-effective adaptation strategy using small scale noisy synthetic data generated by translating English Reddit title-body pairs with open-weights models. We establish a comprehensive evaluation benchmark comprising existing datasets, translated data, and a manually curated dataset. Our experiments reveal a surprising "Less is More" phenomenon: fine-tuning a multilingual encoder (mE5) on just 10,000 noisy synthetic pairs yields 11-12\% average improvements across the benchmark with a 20\%+ relative improvement in retrieval performance, matching the performance of models trained on ~1 million examples. Furthermore, we demonstrate that neither increasing data scale, improving translation quality via state-of-the-art LLMs, nor diversifying data domains yields significant gains over this minimal baseline. We validate the generalizability of these findings on another LRL with a unique script. Our results suggest that semantic alignment for LRLs saturates early and is highly robust to noise, democratizing high-performance embedding creation for resource-constrained communities. We release the model, data, and the benchmark at https://metric-ai-lab.github.io/less-is-more-embeddings/ to facilitate further research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑