arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.09435cs.CL
收藏

MUDIDI:一种基于语言模型的多语言词典数字化两阶段框架

MUDIDI: A Two-Stage Framework for Multilingual Dictionary Digitization with Language Models

School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系统学院);Melbourne School of Psychological Sciences, The University of Melbourne(墨尔本大学墨尔本心理科学学院);LILT

David Setiawan, Temuulen Khishigsuren, Milind Agarwal, Pagnarith Pit, Aso Mahmudi, Ekaterina Vylomova

AI总结:

提出MUDIDI两阶段框架,结合语言模型实现多语言词典数字化,在字符识别、标记保留和词条分割上优于现有OCR和视觉语言模型,并发布30本公共领域词典的标注数据集。

摘要:

多语言词典是低资源和濒危语言最有价值的文献资源之一,但许多仍仅以扫描件形式存在。几十年来,由于语言特有的文字、包含缩写和交叉引用条目的复杂多栏布局,其数字化并转换为机器可读格式几乎不可能。最近的视觉语言模型提供了有希望的解决方案,但尚不清楚它们在保留字符、标记和处理词典结构方面的表现。我们提出MUDIDI,一个用于多语言词典数字化的两阶段框架。第一阶段评估字符识别和标记保留的质量;第二阶段专注于词典条目分割,随后映射到机器可读的词典模式——SIL的多词典格式化器。我们还发布了一个数据集,包含从30本公共领域词典中收集的人工标注的词典条目,这些词典涵盖多种文字系统、语系和格式。我们在该数据集上对OCR系统、通用大语言模型(LLM)和视觉语言模型(VLM)进行了基准测试,展示了LLM在大多数文字系统和语言的两个阶段中的优越性能,并为更具挑战性的场景提供了改进结果的实用指南。最后,我们表明向LLM补充额外信息(如词典引言)可以提高数字化词典的质量。Github: https://github.com/DavidSamuell/MUDIDI-Pipeline-for-Digitization-of-Multilingual-Dictionary/

英文摘要:

Multilingual dictionaries are among the most valuable documentary resources for low-resource and endangered languages, yet many remain available only as scans. For many decades, their digitization and conversion into a machine-readable format was nearly impossible due to language-specific scripts, complex multi-column layouts full of entries with abbreviations and cross-references. Recent vision-language models offer a promising solution, but it is unclear how well they preserve characters, markup, and process lexicographic structure. We introduce MUDIDI, a two-stage framework for multi-lingual dictionary digitization. Stage One evaluates the quality of character recognition and markup preservation; Stage Two focuses on dictionary entry segmentation with subsequent mapping into a machine-readable lexicographic schema, SIL's Multi-Dictionary Formatter. We also release a dataset that consists of human-annotated lexicographic entries collected from 30 public-domain dictionaries featuring diverse writing systems, language families, and formats. We benchmark OCR systems, general-purpose Large Language Models (LLMs), and Vision Language Models (VLMs) on the dataset, demonstrating superior performance of LLMs across most writing systems and languages in both stages, and provide practical guidelines on improving the results for more challenging scenarios. Finally, we show that supplementing additional information, such as dictionary introduction, to the LLMs can improve the quality of the digitized dictionary. Github: https://github.com/naarm-nlp/mudidi

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑