使用 FlanT5-XXL 对零样本立场检测进行基准测试：从训练数据、提示和解码策略洞察其接近 SoTA 的性能
Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance
AI总结：
本文用 FlanT5-XXL 在多个推文立场数据集上评估零样本立场检测，分析提示、解码、困惑度及否定表述的影响，证明其可匹敌微调 SoTA，并发现正向偏差。
摘要：
我们研究了基于 LLM 的零样本立场检测在推文上的性能。我们采用 FlanT5-XXL（一种经过指令微调的开源 LLM），并使用 SemEval 2016 Tasks 6A、6B 以及 P-Stance 数据集，考察其在不同提示与解码策略下的性能及变化，以及该模型的潜在偏差。我们表明，这种零样本方法能够匹敌甚至超越包括微调模型在内的当前最优基准。我们给出了关于其性能的多种洞察，包括对指令和提示的敏感性、解码策略、提示的困惑度，以及对提示中否定和对立表述的敏感性。最后，我们确认该 LLM 未在测试数据集上训练过，并识别出一种正向偏差，该偏差或可部分解释不同解码策略之间的性能差异。
英文摘要：
We investigate the performance of LLM-based zero-shot stance detection on tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and its variations under different prompts and decoding strategies, as well as the potential biases of the model. We show that the zero-shot approach can match or outperform state-of-the-art benchmarks, including fine-tuned models. We provide various insights into its performance including the sensitivity to instructions and prompts, the decoding strategies, the perplexity of the prompts, and to negations and oppositions present in prompts. Finally, we ensure that the LLM has not been trained on test datasets, and identify a positivity bias which may partially explain the performance differences across decoding strategie
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
