向右看有时是对的：探究仅解码器大语言模型在序列标注任务上的能力
Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling
AI总结：
本文针对仅解码器大语言模型在序列标注任务上因因果掩码导致性能受限的问题，提出在微调时逐层移除因果掩码的方法，使其在信息抽取任务上匹配或超越最先进模型。
摘要：
基于掩码语言建模（MLM）的预训练语言模型在自然语言理解（NLU）任务中表现优异。虽然经过微调的基于MLM的编码器在同等规模下持续优于因果语言建模解码器，但最近的仅解码器大语言模型（LLMs）的表现已能与规模较小的基于MLM的编码器相当。尽管其性能随规模扩大而提升，LLMs在信息抽取（IE）任务中仍难以达到最先进的结果，而许多IE任务被形式化为序列标注（SL）。我们假设LLMs在SL上表现不佳源于因果掩码，它阻止模型关注当前token右侧的token。然而，LLMs在SL上的性能究竟如何以及在多大程度上可以得到改善仍不清楚。我们探索了通过在LLM微调过程中逐层移除因果掩码（CM）来提升开放LLMs在IE任务上的SL性能的技术。该方法带来的性能提升可与最先进的SL模型相竞争，匹配或超过了从所有块中移除CM的结果。我们的发现在多种SL任务上均成立，表明采用逐层依赖的CM移除的开放LLMs优于强大的基于MLM的编码器，甚至优于经过指令微调的LLMs。
英文摘要：
Pre-trained language models based on masked language modeling (MLM) excel in natural language understanding (NLU) tasks. While fine-tuned MLM-based encoders consistently outperform causal language modeling decoders of comparable size, recent decoder-only large language models (LLMs) perform on par with smaller MLM-based encoders. Although their performance improves with scale, LLMs fall short of achieving state-of-the-art results in information extraction (IE) tasks, many of which are formulated as sequence labeling (SL). We hypothesize that LLMs' poor SL performance stems from causal masking, which prevents the model from attending to tokens on the right of the current token. Yet, how exactly and to what extent LLMs' performance on SL can be improved remains unclear. We explore techniques for improving the SL performance of open LLMs on IE tasks by applying layer-wise removal of the causal mask (CM) during LLM fine-tuning. This approach yields performance gains competitive with state-of-the-art SL models, matching or outperforming the results of CM removal from all blocks. Our findings hold for diverse SL tasks, demonstrating that open LLMs with layer-dependent CM removal outperform strong MLM-based encoders and even instruction-tuned LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
