arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08951cs.CL
收藏

我们该如何建模语言的概率?

How Should We Model the Probability of a Language?

Inria(法国国家信息与自动化研究所);Common Crawl Foundation(Common Crawl基金会)

Rasul Dent, Pedro Ortiz Suarez, Thibault Clérice, Benoît Sagot

AI总结:

本文主张将语言识别从脱离语境的文本分类转向路由问题,以改进尾部语言的覆盖范围。

摘要:

世界上有超过7000种语言被使用,但商业语言识别(LID)系统只能可靠地识别书面形式中的一百多种。研究级系统在某些情况下可以扩展这种覆盖范围,但大多数语言的覆盖仍然零散或不存在。本文主张这种状况主要是自我造成的。特别是,它源于将LID视为脱离语境的文本分类的持续性框架,这种框架掩盖了先验概率估计的核心作用,并受到制度激励的影响,这些激励倾向于支持全球、固定先验模型。我们主张,通过将LID重新视为路由问题,并开发有原则的方法来整合使语言在本地合理的环境线索,来提高尾部语言的覆盖范围。

英文摘要:

Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑