创造性变异性的几何：Credal 集如何揭示语言模型中的校准差距
The Geometry of Creative Variability: How Credal Sets Expose Calibration Gaps in Language Models
AI总结：
该研究利用 credal 集几何框架分解文本生成的认知性与偶然性不确定性，通过大规模故事生成实验揭示语言模型与人类创造性变化之间的校准差距。
摘要：
理解大型语言模型中的不确定性仍是一项根本性挑战，尤其是在存在多个有效输出的创造性任务中。我们提出了一个使用 credal 集（概率分布的凸包）的几何框架，用于量化并分解神经文本生成中的不确定性，并依据人类创造性变化进行校准。我们分析了 WritingPrompts 数据集中的 500 个创意写作提示，每个提示配有 10 段独特的人类续写，并在五种解码策略下评估四个语言模型，共生成 100,000 个故事。我们的 credal 集分析揭示，模型在捕捉人类创造性变化方面存在显著差距，最佳的模型—人类校准仅达到 0.434（采用温度 0.7 的 Gemma-2B）。我们将总不确定性分解为认知性和偶然性成分，发现解码策略的选择贡献了总认知不确定性的 39.4% 至 72.0%。模型规模与校准质量呈弱相关，并且基础模型与指令微调模型在校准质量上不存在显著差异。我们的几何框架为改进生成系统以实现人类—AI 创造性对齐提供了可操作的洞见。我们公开了完整的实验框架。
英文摘要：
Understanding uncertainty in large language models remains a fundamental challenge, particularly in creative tasks where multiple valid outputs exist. We present a geometric framework using credal sets - convex hulls of probability distributions - to quantify and decompose uncertainty in neural text generation, calibrated against human creative variation. Analyzing 500 creative writing prompts from the WritingPrompts dataset with 10 unique human continuations each, we evaluate four language models across five decoding strategies, generating 100,000 stories. Our credal set analysis reveals substantial gaps in capturing human creative variation, with the best model-human calibration reaching only 0.434 (Gemma-2B with temperature 0.7). We decompose total uncertainty into epistemic and aleatoric components, finding that the choice of decoding strategy contributes 39.4% to 72.0% of total epistemic uncertainty. Model scale shows weak correlation with calibration quality and no significant difference exists between base and instruction-tuned models in calibration quality. Our geometric framework provides actionable insights for improving generation systems for human-AI creative alignment. We release our complete experimental framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
