arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.06387cs.CLcs.LG
收藏

序贯预测算法

Rationales for Sequential Predictions

Columbia University(哥伦比亚大学);Harvard University(哈佛大学);Cornell Tech(康奈尔科技校区)

Keyon Vafa, Yuntian Deng, David M. Blei, Alexander M. Rush

摘要:

序列模型是现代NLP系统的重要组成部分,但它们的预测很难解释。我们认为模型的解释,通过理据,上下文的子集,可以解释个别模型的预测。我们通过求解一个组合优化来找到序列的基本原理:最好的基本原理是预测与完整序列相同的输出的输入令牌的最小子集。枚举所有子集是棘手的,因此我们提出了一种高效的贪心算法来逼近这一目标。该算法,这是所谓的贪婪合理化,适用于任何模型。为了使这种方法有效,当对上下文的不完整子集进行预测时,模型应该形成兼容的条件分布。这个条件可以通过一个短的微调步骤来实现。本文研究了贪婪合理化在语言建模和机器翻译中的应用。与现有的基线相比,贪婪合理化是最好的优化组合目标,并提供最忠实的理由。在一个新的数据集的注释顺序的理由,贪婪的理由是最相似的人类的理由。

英文摘要:

Sequence models are a critical component of modern NLP systems, but their predictions are difficult to explain. We consider model explanations though rationales, subsets of context that can explain individual model predictions. We find sequential rationales by solving a combinatorial optimization: the best rationale is the smallest subset of input tokens that would predict the same output as the full sequence. Enumerating all subsets is intractable, so we propose an efficient greedy algorithm to approximate this objective. The algorithm, which is called greedy rationalization, applies to any model. For this approach to be effective, the model should form compatible conditional distributions when making predictions on incomplete subsets of the context. This condition can be enforced with a short fine-tuning step. We study greedy rationalization on language modeling and machine translation. Compared to existing baselines, greedy rationalization is best at optimizing the combinatorial objective and provides the most faithful rationales. On a new dataset of annotated sequential rationales, greedy rationales are most similar to human rationales.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑