魔鬼在细节中：评估基于transformer的方法对粒度任务的限制
The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks
摘要：
近年来，从基于transformer的神经语言模型导出的上下文嵌入在诸如问答、情感分析和文本相似性等各种任务中表现出了最先进的性能。大量的工作表明，这些模型可以准确地表示文本中存在的抽象语义信息。在这个短暂的工作中，我们探索了切线方向，并分析了这些模型在需要更细粒度表示的任务上的性能。我们专注于从两个角度的文本相似性的问题：在粒度级别上匹配文档（需要嵌入捕获文本中的细粒度属性），和一个抽象的水平（需要嵌入捕获整体的文本语义）。我们通过经验证明，在来自不同领域的两个数据集上，尽管抽象文档匹配的性能如预期的那样高，但对于更细粒度的任务，上下文嵌入始终（有时，大大）优于TF-IDF等简单基线。然后，我们提出了一个简单但有效的方法，将TF-IDF模型，使用上下文嵌入，实现了高达36%的粒度任务的相对改善。
英文摘要：
Contextual embeddings derived from transformer-based neural language models have shown state-of-the-art performance for various tasks such as question answering, sentiment analysis, and textual similarity in recent years. Extensive work shows how accurately such models can represent abstract, semantic information present in text. In this expository work, we explore a tangent direction and analyze such models' performance on tasks that require a more granular level of representation. We focus on the problem of textual similarity from two perspectives: matching documents on a granular level (requiring embeddings to capture fine-grained attributes in the text), and an abstract level (requiring embeddings to capture overall textual semantics). We empirically demonstrate, across two datasets from different domains, that despite high performance in abstract document matching as expected, contextual embeddings are consistently (and at times, vastly) outperformed by simple baselines like TF-IDF for more granular tasks. We then propose a simple but effective method to incorporate TF-IDF into models that use contextual embeddings, achieving relative improvements of up to 36% on granular tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
