探究基于Transformer的RDF到文本模型中的遗漏与失真
Probing Omissions and Distortions in Transformer-based RDF-to-Text Models
AI总结:
针对NLG中信息遗漏问题,本文提出两种探测方法分析BART和T5编码器输出,发现编码器对遗漏和失真实体信号较弱导致信息丢失,证明探测法可检测模型输出错误。
摘要:
在自然语言生成(NLG)中,输出文本有时会遗漏重要信息。为了更好地理解和分析此类错误是如何产生的,我们聚焦于RDF到文本生成,并探索了两种探究BART(Lewis等,2020)和T5(Raffel等,2019)编码器输出中信息遗漏的方法:(i)一种新颖的无参数探测方法,基于计算RDF图与移除了部分实体的RDF图嵌入之间的余弦相似度;(ii)一种参数化探测方法,对编码器嵌入执行二分类以检测被遗漏的实体。我们还将分析扩展至失真实体,即在生成文本中未被完全正确提及的实体(例如实体拼写错误、错误的测量单位)。我们发现,遗漏和失实的实体均可在编码器的输出嵌入中被探测到。这表明编码器对这些实体发出了较弱的信号,因此应对部分信息丢失负责。这也表明探测方法可用于检测NLG模型输出中的错误。
英文摘要:
In Natural Language Generation (NLG), important information is sometimes omitted in the output text. To better understand and analyse how this type of mistake arises, we focus on RDF-to-Text generation and explore two methods of probing omissions in the encoder output of BART (Lewis et al, 2020) and of T5 (Raffel et al, 2019): (i) a novel parameter-free probing method based on the computation of cosine similarity between embeddings of RDF graphs and of RDF graphs in which we removed some entities and (ii) a parametric probe which performs binary classification on the encoder embeddings to detect omitted entities. We also extend our analysis to distorted entities, i.e. entities that are not fully correctly mentioned in the generated text (e.g. misspelling of entity, wrong units of measurement). We found that both omitted and distorted entities can be probed in the encoder's output embeddings. This suggests that the encoder emits a weaker signal for these entities and therefore is responsible for some loss of information. This also shows that probing methods can be used to detect mistakes in the output of NLG models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。