A key component of in-context reasoning is the ability of language models (LMs) to bind entities for later retrieval. For example, an LM might represent "Ann loves pie" by binding "Ann" to "pie", allowing it to later retrieve "Ann" when asked "Who loves pie?" Prior research on short lists of bound entities found strong evidence that LMs implement such retrieval via a positional mechanism, where "Ann" is retrieved based on its position in context. In this work, we find that this mechanism generalizes poorly to more complex settings; as the number of bound entities in context increases, the positional mechanism becomes noisy and unreliable in middle positions. To compensate for this, we find that LMs supplement the positional mechanism with a lexical mechanism (retrieving "Ann" using its bound counterpart "pie") and a reflexive mechanism (retrieving "Ann" through a direct pointer). Through extensive experiments on nine models and ten binding tasks, we uncover a consistent pattern in how LMs mix these mechanisms to drive model behavior. We leverage these insights to develop a causal model combining all three mechanisms that estimates next token distributions with 95% agreement. Finally, we show that our model generalizes to substantially longer inputs of open-ended text interleaved with entity groups, further demonstrating the robustness of our findings in more natural settings. Overall, our study establishes a more complete picture of how LMs bind and retrieve entities in-context.
Self-Adaptive Named Entity Recognition by Retrieving Unstructured Knowledge
NTT Human Informatics Laboratories, NTT Corporation(NTT公司NTT人类信息学实验室);The University of Tokyo(东京大学);Institute of Industrial Science, the University of Tokyo(东京大学生产技术研究所)
针对目标领域 NER 依赖昂贵训练数据或知识库的问题,提出以低置信实体为查询检索非结构化知识并修正预测的两阶段自适应模型,在 CrossNER 上 F1 提升 2.35 分。
Improving Few-shot and Zero-shot Entity Linking with Coarse-to-Fine Lexicon-based Retriever
Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(广东省新型安全智能技术重点实验室);Central South University(中南大学)
Recall, Retrieve and Reason: Towards Better In-Context Relation Extraction
School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Beijing Institute of Computer Technology and Application(北京计算机技术及应用研究所);Interdisciplinary Applications (Southeast University), Ministry of Education(教育部东南大学交叉应用中心)
Mixing Context Granularities for Improved Entity Linking on Question Answering Data across Entity Categories
Ubiquitous Knowledge Processing Lab (UKP)(普适知识处理实验室(UKP));Research Training Group AIPHES(AIPHES研究培训小组);Department of Computer Science, Technische Universität Darmstadt(达姆施塔特工业大学计算机科学系)
Borrowing or Codeswitching? Annotating for Finer-Grained Distinctions in Language Mixing
UNED(西班牙国立远程教育大学);Brandeis University(布兰迪斯大学)
我们提出了一个新的语料库的Twitter数据注释代码转换和借用西班牙语和英语之间。该语料库包含9,500条tweet,这些tweet在token级别上使用代码转换,借用和命名实体进行注释。本语料库与以往的语码转换语料库不同之处在于,我们试图清楚地定义和注释语码转换和借用之间的界限,而不处理常见的“网络用语”(“lol”等)。在单语语境中使用时被认为是语码转换。其结果是一个语料库,使研究和建模的西班牙语-英语借用和代码转换在Twitter上的一个数据集。我们使用基于transformer的语言模型为该语料库的标签建模提供了基线分数。注释本身是以CC BY 4.0许可证发布的,而它所适用的文本是按照Twitter服务条款分发的。
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
Studio Ousia;RIKEN AIP(理化学研究所先进智能项目中心);University of Washington(华盛顿大学);Nara Institute of Science and Technology(奈良先端科学技术大学院大学);National Institute of Informatics(国立情报学研究所)
Prompting Large Language Models with Partial Knowledge for Answering Questions with Unseen Entities
School of Computer and Information Technology, Shanxi University(山西大学计算机与信息学院);Beijing University of Technology(北京理工大学);Department of Computer Science, University of Manchester(曼彻斯特大学计算机科学系);Singapore University of Technology and Design(新加坡科技与设计大学);ILCC, School of Informatics, University of Edinburgh(爱丁堡大学信息学院ILCC)
ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models
School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Beijing Institute of Computer Technology and Application(北京计算机技术及应用研究所);Inspur Group Co., Ltd.(浪潮集团有限公司);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
DAMO-NLP at SemEval-2022 Task 11: A Knowledge-based System for Multilingual Named Entity Recognition
Zhejiang University(浙江大学);Singapore University of Technology and Design(新加坡科技设计大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院);ShanghaiTech University(上海科技大学)