外部信息对基于大语言模型的立场检测是否有用？
Is External Information Useful for Stance Detection with LLMs?
AI总结：
本文系统评估了维基百科和网络搜索外部信息对八个大语言模型在三个数据集上立场检测的影响，发现外部信息多数情况下反而降低性能，并揭示其源于模型偏向所提供信息的立场与情感而非文本真实立场。
摘要：
在立场检测任务中，一段文本被分类为对某一目标持支持、反对或中立态度。此前的研究表明，使用外部信息（例如来自维基百科的摘录）能够提升立场检测的性能。然而，尽管大语言模型（LLMs）已在众多推理任务中被广泛采用，此类信息是否能为大语言模型带来益处仍是一个尚未解答的问题。在本研究中，我们对维基百科和网络搜索外部信息如何影响立场检测进行了系统性评估，覆盖八个大语言模型以及包含12个目标的三个数据集。令人意外的是，我们发现此类信息在大多数情况下会降低性能，宏观F1分数最多下降27.9%。我们通过实验对此进行解释，实验表明大语言模型倾向于使其预测与所提供信息的立场和情感保持一致，而非与给定文本的真实立场保持一致。我们还发现，即使采用思维链提示，性能下降仍然存在，而微调可以缓解但无法完全消除这种下降。与先前基于BERT系统的文献认为外部信息能提升性能的结论相反，我们的研究结果凸显了基于大语言模型的立场分类器中信息偏差所带来的风险。代码可在 https://github.com/ngqm/acl2025-stance-detection 获取。
英文摘要：
In the stance detection task, a text is classified as either favorable, opposing, or neutral towards a target. Prior work suggests that the use of external information, e.g., excerpts from Wikipedia, improves stance detection performance. However, whether or not such information can benefit large language models (LLMs) remains an unanswered question, despite their wide adoption in many reasoning tasks. In this study, we conduct a systematic evaluation on how Wikipedia and web search external information can affect stance detection across eight LLMs and in three datasets with 12 targets. Surprisingly, we find that such information degrades performance in most cases, with macro F1 scores dropping by up to 27.9\%. We explain this through experiments showing LLMs' tendency to align their predictions with the stance and sentiment of the provided information rather than the ground truth stance of the given text. We also find that performance degradation persists with chain-of-thought prompting, while fine-tuning mitigates but does not fully eliminate it. Our findings, in contrast to previous literature on BERT-based systems which suggests that external information enhances performance, highlight the risks of information biases in LLM-based stance classifiers. Code is available at https://github.com/ngqm/acl2025-stance-detection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
