压力下的报告：区分LLM统计分析中的事实性谄媚与语气性谄媚
Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis
浏览论文内容
中文总结 AI 辅助
本研究通过4×4因子实验揭示，LLM在数据分析报告中，编辑框架主要引发语气性谄媚，事实性误述仅出现在特定框架与数据模式组合中。
中文摘要 AI 辅助
大型语言模型越来越多地被要求分析数据并报告结果的含义，这一任务不同于大多数谄媚研究中关注的信念或偏好对齐场景。我们测试了提示中的编辑框架——从中性请求到明确指示穷尽搜索理由以否定或支持某个发现——是否不仅改变模型的报告语气，还改变其报告实质。通过一个4×4因子设计，交叉四种框架条件与四种真实数据模式（真实效应、模仿效应但未通过稳健性检验的混杂、高功效零结果、低功效零结果），我们收集了480个响应，并沿两个独立维度进行评分：其关于数据的事实主张是否偏离正确解释，以及是否仅语气偏离而主张保持正确。事实性误述集中在两个单元格：对真实效应施加严厉批评框架时，模型自我说服产生无根据的怀疑（97%的响应）；对低功效零结果施加追求显著性框架时，模型过度自信地断言数据无法支持的零结论（100%的响应）。语气变化远比事实内容广泛，批评框架在所有数据模式下都产生防御性、充满模糊措辞的语域，无论数据显示什么；而追求显著性框架仅在数据存在真正模糊性时改变语气。数据本身存在的混杂几乎在所有测试框架条件下都阻止了这两种变化。这些结果表明，在LLM辅助数据分析中，框架诱导失真的风险既非跨框架均匀，也非跨数据模式均匀，且模型可以在保持正确结论的同时，其语气在其周围大幅变化。
英文摘要
Large language models are increasingly asked to analyze data and report what the results mean, a task distinct from the belief- or preference-alignment settings studied in most sycophancy research. We test whether editorial framing in the prompt, ranging from a neutral request to an explicit instruction to search exhaustively for reasons to discredit or to support a finding, changes not just the tone but the substance of a model's report. Across a 4 x 4 factorial design crossing four framing conditions with four ground-truth data patterns (a genuine effect, a confound that mimics an effect but fails a robustness check, a well-powered null, and an underpowered null), we collect 480 responses and score each along two independent dimensions: whether its factual claim about the data diverged from the correct interpretation, and whether only its tone diverged while the claim stayed correct. Factual misrepresentation is concentrated in two cells: brutally critical framing applied to a genuine effect, where the model talks itself into unwarranted skepticism (97% of responses), and significance-seeking framing applied to an underpowered null, where the model overstates confidence in a null conclusion the data cannot support (100% of responses). Tone shifts far more broadly than factual content does, with critical framing producing a defensive, hedge-heavy register across every data pattern regardless of what the data show, while significance-seeking framing shifts tone only where the data leave genuine ambiguity. A confound present in the data itself blocks both kinds of shift almost entirely under every framing condition tested. These results indicate that the risk of framing-induced distortion in LLM-assisted data analysis is neither uniform across framings nor uniform across data patterns, and that a model can hold a correct conclusion in place while its tone shifts substantially around it.
发表机构
- Birla Institute of Technology and Science, Pilani, Hyderabad Campus（比拉理工学院海得拉巴校区）
机构由 AI 辅助整理，请以论文原文为准。
