arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.14675cs.CL
收藏

众包皮埃蒙特语以测试LLM在非标准正写法上的表现

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics(查理大学数学与物理学院形式与应用语言学研究所)

Gianluca Vico, Jindřich Libovický

AI总结:

通过众包皮埃蒙特语数据集测试LLM在非标准正写法上的表现,发现分词惩罚但分类性能接近主流语言,翻译结果存在不对称性。

摘要:

我们提出了一个众包数据集,用于皮埃蒙特语,这是一种意大利西北部的濒危罗曼语。该数据集包含145个意大利-皮埃蒙特语平行句子,源自Flores+,其中翻译由用自然正写法书写的人完成,而非遵循标准化规范,同时还包含人工词对齐。我们利用此资源对几种大型语言模型进行基准测试,测试内容包括分词平衡、主题分类和机器翻译。我们的分析表明,皮埃蒙特语相对于资源更丰富的罗曼语存在分词惩罚,但LLM在分类性能上接近意大利、法语和英语。机器翻译结果是不对称的:模型能很好地将皮埃蒙特语翻译成高资源语言,但生成皮埃蒙特语仍具有挑战性。数据集和代码已公开发布。

英文摘要:

We present a crowdsourced dataset for Piedmontese, an endangered Romance language of northwestern Italy. The dataset comprises 145 Italian-Piedmontese parallel sentences derived from Flores+, with translations produced by speakers writing in their natural orthographic style rather than adhering to standardized conventions, along with manual word alignment. We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation. Our analysis reveals that Piedmontese incurs a tokenization penalty relative to higher-resource Romance languages, yet LLMs achieve classification performance approaching that of Italian, French, and English. Machine translation results are asymmetric: models translate adequately from Piedmontese into high-resource languages, but generation into Piedmontese remains challenging. The dataset and code are publicly released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑