arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.11504cs.CL
收藏

语言模型行为:一项全面综述

Language Model Behavior: A Comprehensive Survey

UC San Diego(加州大学圣迭戈分校)

Tyler A. Chang, Benjamin K. Bergen

AI总结:

该综述梳理250余项微调前英文Transformer语言模型行为研究,分析其基础能力的敏感性与现存弱点的成因,为相关应用及跨领域研究提供参考。

摘要:

Transformer语言模型已受到公众广泛关注,但它们生成的文本即便对NLP研究者而言也时常出人意料。本综述讨论了250余项针对任务特定微调前英文语言模型行为的近期研究。语言模型在句法、语义、语用、世界知识和推理方面具备基础能力,但这些能力对特定输入和表面特征较为敏感。尽管模型规模扩展至数千亿参数时,生成文本的质量有了显著提升,但模型仍容易出现非事实性回复、常识错误、记忆文本泄露以及社会偏见问题。这些弱点中有许多可被界定为对文本中习得模式的过度泛化或泛化不足。我们综合了近期研究成果,明确了当前关于大语言模型能力的已知认知,从而为应用工作及使用语言模型的相邻领域研究提供参考资源。

英文摘要:

Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about large language model capabilities, thus providing a resource for applied work and for research in adjacent fields that use language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑