arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2108.04106cs.CLcs.AI
收藏

Few-Shot文本分类的噪声通道语言模型建模

Noisy Channel Language Model Prompting for Few-Shot Text Classification

Facebook AI Research(脸书人工智能研究院);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)

Sewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer

摘要:

提出了一种基于噪声通道的语言模型提示方法,用于Few-Shot文本分类。通道模型不是计算给定输入的标签的似然性(称为直接模型),而是计算给定标签的输入的条件概率,从而需要解释输入中的每个单词。我们使用通道模型最近提出的几次少样本学习方法,没有或非常有限的更新语言模型参数,通过在上下文演示或提示微调。我们的实验表明,对于这两种方法,信道模型明显优于其直接对应模型,我们将其归因于它们的稳定性,即,更低的方差和更高的最差情况准确度。我们还提供了广泛的消融,为何时使用通道提示微调而不是其他竞争性方法(例如,直接头部调谐):当训练样本的数量较少、训练数据中的标签不平衡或者需要对看不见的标签进行概括时,优选信道提示微调。

英文摘要:

We introduce a noisy channel approach for language model prompting in few-shot text classification. Instead of computing the likelihood of the label given the input (referred as direct models), channel models compute the conditional probability of the input given the label, and are thereby required to explain every word in the input. We use channel models for recently proposed few-shot learning methods with no or very limited updates to the language model parameters, via either in-context demonstration or prompt tuning. Our experiments show that, for both methods, channel models significantly outperform their direct counterparts, which we attribute to their stability, i.e., lower variance and higher worst-case accuracy. We also present extensive ablations that provide recommendations for when to use channel prompt tuning instead of other competitive methods (e.g., direct head tuning): channel prompt tuning is preferred when the number of training examples is small, labels in the training data are imbalanced, or generalization to unseen labels is required.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑