面向反事实生成的大语言模型提示方法：一项实证研究
Prompting Large Language Models for Counterfactual Generation: An Empirical Study
AI总结：
本文通过构建覆盖多种NLU任务的评估框架，实证研究了大语言模型生成反事实的能力及其影响因素，发现对齐技术可提升该能力而单纯增大模型规模无效，并揭示了提示设计中任务指南与思维链方法的不同作用。
摘要：
大语言模型（LLMs）在广泛的自然语言理解与生成任务中取得了显著进展。然而，它们生成反事实的能力尚未得到系统性检验。为弥补这一空白，我们提出了一个覆盖多种自然语言理解（NLU）任务类型的综合评估框架，该框架涵盖了决定LLMs反事实生成能力的所有关键因素。基于该框架，我们1）考察了LLMs作为反事实生成器的优势与不足，2）揭示了影响LLMs生成反事实的因素，包括LLMs的内在属性与提示设计。结果表明，尽管LLMs在大多数情况下表现良好，但它们在关系抽取（RE）等复杂任务中面临挑战，因为其受限于任务特定性能、实体约束和固有的选择偏差。我们还发现，对齐技术，例如指令微调和基于人类反馈的强化学习，可能潜在地增强LLMs的反事实生成能力。相反，单纯增加参数规模并不能带来预期的改进。此外，从提示设计的角度来看，任务指南毫不意外地发挥着重要作用。然而，思维链方法由于一致性问题并不总能提供帮助。
英文摘要：
Large language models (LLMs) have made remarkable progress in a wide range of natural language understanding and generation tasks. However, their ability to generate counterfactuals has not been examined systematically. To bridge this gap, we present a comprehensive evaluation framework on various types of NLU tasks, which covers all key factors in determining LLMs' capability of generating counterfactuals. Based on this framework, we 1) investigate the strengths and weaknesses of LLMs as the counterfactual generator, and 2) disclose the factors that affect LLMs when generating counterfactuals, including both the intrinsic properties of LLMs and prompt designing. The results show that, though LLMs are promising in most cases, they face challenges in complex tasks like RE since they are bounded by task-specific performance, entity constraints, and inherent selection bias. We also find that alignment techniques, e.g., instruction-tuning and reinforcement learning from human feedback, may potentially enhance the counterfactual generation ability of LLMs. On the contrary, simply increasing the parameter size does not yield the desired improvements. Besides, from the perspective of prompt designing, task guidelines unsurprisingly play an important role. However, the chain-of-thought approach does not always help due to inconsistency issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
