LLM 生成合成数据在在线政治讨论立场检测中的力量
The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions
AI总结：
本文提出利用 LLM 离线生成合成数据并结合主动学习筛选最具信息量样本，以少量标注数据提升在线政治讨论中立场检测模型的性能，超越全量标注微调的基线模型。
摘要：
立场检测在改善在线政治讨论方面具有巨大潜力，可通过将其部署于讨论平台来实现内容审核、主题摘要或促进更均衡的讨论等目的。通常，基于 Transformer 的模型被直接用于立场检测，这需要大量数据。然而，在线政治讨论中辩论主题的广泛多样性使得数据收集尤为困难。LLM 已使立场检测重新焕发活力，但其在在线政治讨论中的在线部署面临输出不一致、偏见以及易受对抗攻击等挑战。我们展示了 LLM 生成的合成数据如何改善在线政治讨论的立场检测：在在线部署中使用可靠的传统立场检测模型，同时在安全的离线环境中利用 LLM 的文本生成能力进行合成数据生成。为实现这一目标，（i）我们通过提示 Mistral-7B 模型为特定辩论问题生成合成数据，并表明使用生成的合成数据进行微调可以显著提升立场检测的性能，同时保持可解释性并与真实世界数据保持一致。（ii）以合成数据作为参考，我们通过识别未标注数据集中最具信息量的样本——即立场检测模型最不确定且能从中获益最多的样本——可以进一步提升性能。通过同时使用合成数据和最具信息量的样本进行微调，我们在标注数据量显著减少的情况下，超越了在所有真实标签上微调的基线模型的性能。
英文摘要：
Stance detection holds great potential to improve online political discussions through its deployment in discussion platforms for purposes such as content moderation, topic summarization or to facilitate more balanced discussions. Typically, transformer-based models are employed directly for stance detection, requiring vast amounts of data. However, the wide variety of debate topics in online political discussions makes data collection particularly challenging. LLMs have revived stance detection, but their online deployment in online political discussions faces challenges like inconsistent outputs, biases, and vulnerability to adversarial attacks. We show how LLM-generated synthetic data can improve stance detection for online political discussions by using reliable traditional stance detection models for online deployment, while leveraging the text generation capabilities of LLMs for synthetic data generation in a secure offline environment. To achieve this, (i) we generate synthetic data for specific debate questions by prompting a Mistral-7B model and show that fine-tuning with the generated synthetic data can substantially improve the performance of stance detection, while remaining interpretable and aligned with real world data. (ii) Using the synthetic data as a reference, we can improve performance even further by identifying the most informative samples in an unlabelled dataset, i.e., those samples which the stance detection model is most uncertain about and can benefit from the most. By fine-tuning with both synthetic data and the most informative samples, we surpass the performance of the baseline model that is fine-tuned on all true labels, while labelling considerably less data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
