arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.19645cs.CL
收藏

信号即是上限:从开放式调查文本中LLM预测经验评分的测量限制

The signal is the ceiling: Measurement limits of LLM-predicted experience ratings from open-ended survey text

Dimension Labs(Dimension实验室)

Andrew Hong, Jason Potteiger, Luis E. Zapata

AI总结:

本文研究了提示设计和模型选择对LLM预测经验评分的影响,发现文本本身的语言特征对准确性影响更大,提示工程只能在特定范围内提升性能。

摘要:

一篇早期论文(Hong, Potteiger, and Zapata 2026)表明,未经优化的GPT 4.1提示在67%的情况下能从开放式调查文本中预测粉丝报告的经验评分。本文测试了提示设计和模型选择对性能的相对影响。我们比较了四个配置,涉及约10,000份赛后调查问卷:原始基线提示和一个中等定制版本,结合三个GPT模型(4.1、4.1-mini、5.2)。提示定制使GPT 4.1的±1内一致率提高了约2个百分点(从67%到69%)。从最佳配置切换模型会降低性能:GPT 5.2回到基线,GPT 4.1-mini下降了6个百分点。两者结合的效果远不及输入本身:在有能力的配置中,准确性在文本语言特征上的变化比提示或模型的选择大一个数量级。上限有两个部分。一个是模型读取文本的偏差,提示设计可以纠正。另一个是粉丝写的和他们实际决定的差异,工程无法关闭,因为缺失信息不在文本中。提示定制影响了第一部分,模型选择无法可靠地影响。结果不是提示工程帮助一点,而是提示工程在特定且可预测的方式上帮助,仅在它能到达的上限部分。

英文摘要:

An earlier paper (Hong, Potteiger, and Zapata 2026) established that an unoptimized GPT 4.1 prompt predicts fan-reported experience ratings within one point 67% of the time from open-ended survey text. This paper tests the relative impact of prompt design and model selection on that performance. We compared four configurations on approximately 10,000 post-game surveys from five MLB teams: the original baseline prompt and a moderately customized version, crossed with three GPT models (4.1, 4.1-mini, 5.2). Prompt customization added roughly two percentage points of within +/-1 agreement on GPT 4.1 (from 67% to 69%). Both model swaps from that best configuration degraded performance: GPT 5.2 returned to the baseline, and GPT 4.1-mini fell six percentage points below it. Both levers combined were dwarfed by the input itself: across capable configurations, accuracy varied more than an order of magnitude more by the linguistic character of the text than by the choice of prompt or model. The ceiling has two parts. One is a bias in how the model reads text, which prompt design can correct. The other is a difference between what fans write about and what they actually decide, which no engineering can close because the missing information is not in the text. Prompt customization moved the first part; model selection moved neither reliably. The result is not that "prompt engineering helps a little" but that prompt engineering helps in a specific and predictable way, on the part of the ceiling it can reach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑