CONTESTS：语言模型中跨度概率一致性测试框架
CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models
AI总结：
本文提出ConTestS框架，通过统计测试评估语言模型在不同补全与条件顺序下跨度概率的一致性，发现两类模型均存在不一致且趋势相反，并表明预测熵可辅助选择解码策略。
摘要：
尽管语言模型分数常被当作概率使用，但其作为概率估计器的可靠性主要通过校准来研究，忽略了其他方面。特别是，目前尚不清楚语言模型在以不同方式为词跨度分配联合概率时是否会产生相同的值。我们的工作引入了一个新颖的框架 ConTestS（基于跨度的一致性测试），该框架包含统计测试，用于评估可互换的补全顺序和条件顺序下的分数一致性。我们在发布后的真实数据和合成数据上进行实验，以消除训练效应。我们的发现表明，掩码语言模型（MLMs）和自回归模型都表现出不一致的预测，其中自回归模型显示出更大的差异。更大的掩码语言模型倾向于产生更一致的预测，而自回归模型则呈现相反的趋势。此外，对于两种模型类型，预测熵都能为真实词跨度似然提供洞见，因此有助于选择最优解码策略。我们的分析所揭示的不一致性，以及它们与预测熵的关联和模型类型之间的差异，可以作为未来研究解决这些局限性的有用指南。
英文摘要：
Although language model scores are often treated as probabilities, their reliability as probability estimators has mainly been studied through calibration, overlooking other aspects. In particular, it is unclear whether language models produce the same value for different ways of assigning joint probabilities to word spans. Our work introduces a novel framework, ConTestS (Consistency Testing over Spans), involving statistical tests to assess score consistency across interchangeable completion and conditioning orders. We conduct experiments on post-release real and synthetic data to eliminate training effects. Our findings reveal that both Masked Language Models (MLMs) and autoregressive models exhibit inconsistent predictions, with autoregressive models showing larger discrepancies. Larger MLMs tend to produce more consistent predictions, while autoregressive models show the opposite trend. Moreover, for both model types, prediction entropies offer insights into the true word span likelihood and therefore can aid in selecting optimal decoding strategies. The inconsistencies revealed by our analysis, as well their connection to prediction entropies and differences between model types, can serve as useful guides for future research on addressing these limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
