利用语言模型修复古代文档的文本

中文摘要 AI 辅助

目的：本研究探究利用语言模型修复受损古代手稿中因物理缺损造成的缺失文本的可行性。方法：本研究提出不同场景以复现真实情况，根据各场景适用性应用不同架构的语言模型，还提出多种解码策略以进一步提升性能，并解决缺损边界与模型分词方案之间的差异问题。结果：研究结果表明，这类文档的文本修复无法完全自动化，但可作为有用工具辅助古文字学家的工作；模型性能因需修复的文档结构部分以及是否可获取缺失文本的字符长度而存在显著差异。创新性：本研究是首个分析公式化与非公式化内容的模型性能，以及缺损长度感知对手稿修复影响的研究，这两者都是古文字学家手动修复工作中反复出现的挑战；通过系统比较不同模型在不同设置下的性能，并结合定性与定量分析，本研究为开发辅助古文字学家的工具提供了指南。

英文摘要

Purpose - This study investigates the feasibility of restoring missing text caused by physical lacunae in damaged ancient manuscripts using language models. Methodology - The study proposes different scenarios to replicate real-world conditions. Language models of different architectures are applied according to their suitability to each scenario. We also propose several decoding strategies that further enhance performance and address the discrepancy between lacuna boundaries and the models' tokenization schemes. Findings - The results reveal that text restoration of these documents cannot be fully automated, but it can serve as a useful tool to assist paleographers in their work. Model performance varies greatly depending on which structural part of the document needs to be restored and whether the character length of missing text is available. Originality - This is the first study and to analyze model performance on formulaic and non-formulaic content and the impact of lacuna length awareness in manuscript restoration. Both are recurring challenges in paleographers' manual restoration work. Through systematic comparison and both qualitative and quantitative analysis of different models' performance under varying settings, this study offers a guideline for developing assistive tools to support paleographers.

发表机构

• University of Bologna（博洛尼亚大学）
• University of Copenhagen（哥本哈根大学）

机构由 AI 辅助整理，请以论文原文为准。