FramingQA：问题是否塑造答案？衡量组合性框架效应
FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect
- University of Oxford（牛津大学）
- Oxford Internet Institute（牛津互联网研究院）
- Thomson Reuters（汤森路透）
- University of California San Diego（加州大学圣迭戈分校）
- University of Cambridge（剑桥大学）
- Institute of Biomedical Engineering, University of Oxford（牛津大学生物医学工程研究所）
- LMU Munich（慕尼黑大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
FramingQA基准通过三个嵌套层级注入框架偏差，评估九个开源LLM在法律、医学、金融和机器人模拟中的敏感性，发现强逐变体准确性不保证跨措辞鲁棒性。
AI中文摘要：
我们引入了FramingQA，这是一个衡量模型在法律、医学、金融和机器人模拟领域中对问题框架敏感度的基准。大型语言模型（LLMs）在面对与用户隐含立场一致的细微措辞变化时，常常会改变其回答。这可能导致用户得到的建议受到他们提问方式的影响，而非基于底层事实，在高风险领域中，这种后果代价高昂。在现实场景中，专家从业者和非专家用户都经常向LLMs提出包含不完整或误导性假设的问题，因此模型极易受到这些框架的影响。为了测试这一点，我们在三个嵌套层级中注入框架偏差：带有框架偏差的问题措辞（根源层）、在中性问题前添加带有框架偏差的前提（命题层），以及前提与带有框架偏差的问题配对（全局层）。通过评估四个家族中的九个开源模型（3.8B-70B），我们发现，在固定事实信息下，强的逐变体准确性并不能保证在不同措辞的问题之间具有鲁棒性。
英文摘要：
We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations. Large language models (LLMs) often change their responses to subtle rephrasings that align with an implied stance by users. This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts, and the consequences are highly costly in high-stakes domains. Because in the realistic scenarios, both expert practitioners and non-expert users frequently ask LLMs questions containing incomplete or misleading assumptions, models are highly susceptible to those framings. To test this, we inject the framing bias across three nested levels: a framing-biased question phrasing (root), an injected framing-biased premise prepended to a neutral question (propositional), and a premise paired with a framing-biased question (global). Evaluating nine open models (3.8B-70B) across four families, we find that strong per-variant accuracy does not guarantee the robustness across differently phrased questions under the fixed factual information.
