arXiv 2411.190962025-11-10cs.CL

Pralekha：面向印度语言的多语言文档对齐

Pralekha: Cross-Lingual Document Alignment for Indic Languages

Nilekani Centre at AI4Bharat（AI4Bharat的Nilekani中心）；Microsoft（微软）；National Institute of Information and Communications Technology, Japan（日本信息与通信技术国家研究所）；Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）；Indian Institute of Technology, Bombay（印度理工学院孟买分校）

Sanjay Suryanarayanan, Haiyue Song, Mohammed Safi Ur Rahman Khan, Anoop Kunchukuttan, Raj Dabre

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Pralekha基准和文档对齐系数DAC，通过块级匹配实现印度语言与英语的细粒度高效文档对齐，显著提升文档级机器翻译性能。

摘要：

由于现有多语言文档对齐（CLDA）技术的局限性，为文档级机器翻译（MT）挖掘平行文档对仍然具有挑战性。现有方法通常依赖URL等元数据，而这些元数据较为稀缺，或者依赖池化文档表示，无法捕捉细粒度的对齐线索。此外，句子嵌入模型有限的上下文窗口阻碍了它们表示文档级上下文的能力，而基于句子的对齐引入了组合上巨大的搜索空间，导致高昂的计算成本。为了解决印度语言面临的这些挑战，我们引入了Pralekha，一个包含超过300万个对齐文档对的基准，覆盖11种印度语言和英语，其中包括150万个英语-印度语对。此外，我们提出了文档对齐系数（DAC），一种用于细粒度文档对齐的新指标。与基于池化的方法不同，DAC通过匹配较小的块来对齐文档，并将相似度计算为已对齐块数与一对文档中平均块数的比值。内在评估表明，我们基于块的方法在保持有竞争力的性能的同时，速度提高了2-3倍，并且DAC相比基于池化的基线取得了显著提升。外在评估进一步表明，在DAC对齐的文档对上训练的文档级MT模型始终优于使用基线对齐方法的模型。这些结果凸显了DAC在平行文档挖掘中的有效性。数据集和评估框架已公开，以支持进一步研究。

英文摘要：

Mining parallel document pairs for document-level machine translation (MT) remains challenging due to the limitations of existing Cross-Lingual Document Alignment (CLDA) techniques. Existing methods often rely on metadata such as URLs, which are scarce, or on pooled document representations that fail to capture fine-grained alignment cues. Moreover, the limited context window of sentence embedding models hinders their ability to represent document-level context, while sentence-based alignment introduces a combinatorially large search space, leading to high computational cost. To address these challenges for Indic languages, we introduce Pralekha, a benchmark containing over 3 million aligned document pairs across 11 Indic languages and English, which includes 1.5 million English-Indic pairs. Furthermore, we propose Document Alignment Coefficient (DAC), a novel metric for fine-grained document alignment. Unlike pooling-based methods, DAC aligns documents by matching smaller chunks and computes similarity as the ratio of aligned chunks to the average number of chunks in a pair. Intrinsic evaluation shows that our chunk-based method is 2-3x faster while maintaining competitive performance, and that DAC achieves substantial gains over pooling-based baselines. Extrinsic evaluation further demonstrates that document-level MT models trained on DAC-aligned pairs consistently outperform those using baseline alignment methods. These results highlight DAC's effectiveness for parallel document mining. The dataset and evaluation framework are publicly available to support further research.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。