BERTering RAMS:BERT对事件论证了解多少?--对RAMS数据集的研究
BERTering RAMS: What and How Much does BERT Already Know About Event Arguments? -- A Study on the RAMS Dataset
摘要:
使用来自(Clark等人,2019),我们观察到,在RAMS数据集上(Ebner et al.,2020年),BERT的注意力头有适度的,但远远高于机会的能力,发现事件的论点,没有任何培训或领域微调,从17.77%的最低位置到51.61%的最高位置。接下来,我们发现这些头部的线性组合,估计约占可用总事件参数检测监督的11%,可以为某些角色带来更高的性能-最高的两个是受害者(68.29%准确率)和受害者(58.82%准确率)。此外,我们调查如何以及我们的方法做跨句子事件参数。我们提出了一个程序,以隔离“最好的头”跨句参数检测单独的那些句内参数。这样估计的中心语与它们的联合估计的等价物相比具有更好的跨句性能,尽管只是在我们已经知道该论点存在于另一个句子中的不切实际的假设下。最后,我们试图隔离我们的数字在多大程度上源于基于词频的黄金论点和角色之间的关联。我们提出了NONCE,一个通过用随机生成的“nonce”单词替换黄金参数来创建对抗性测试示例的方案。我们发现,学习的线性组合是强大的NONCE,虽然个人最好的头可以更敏感。
英文摘要:
Using the attention map based probing frame-work from (Clark et al., 2019), we observe that, on the RAMS dataset (Ebner et al., 2020), BERT's attention heads have modest but well above-chance ability to spot event arguments sans any training or domain finetuning, vary-ing from a low of 17.77% for Place to a high of 51.61% for Artifact. Next, we find that linear combinations of these heads, estimated with approx 11% of available total event argument detection supervision, can push performance well-higher for some roles - highest two being Victim (68.29% Accuracy) and Artifact(58.82% Accuracy). Furthermore, we investigate how well our methods do for cross-sentence event arguments. We propose a procedure to isolate "best heads" for cross-sentence argument detection separately of those for intra-sentence arguments. The heads thus estimated have superior cross-sentence performance compared to their jointly estimated equivalents, albeit only under the unrealistic assumption that we already know the argument is present in an-other sentence. Lastly, we seek to isolate to what extent our numbers stem from lexical frequency based associations between gold arguments and roles. We propose NONCE, a scheme to create adversarial test examples by replacing gold arguments with randomly generated "nonce" words. We find that learnt linear combinations are robust to NONCE, though individual best heads can be more sensitive.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。