正交约束满足与大语言模型中的人类难度对齐
Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models
AI总结:
本文评估了三种大语言模型在字符级约束满足任务上的表现,发现跨家族性能差异显著大于同家族参数扩展效果,且模型在处理常见词时存在系统性失败,揭示了对分布合理性依赖的问题。
摘要:
大型语言模型在受控文本生成中必须满足严格的正交约束,但系统性的跨家族评估仍有限。我们评估了39种配置,涵盖三个模型家族(Qwen3、Claude Haiku 4.5、GPT-5-mini)在58个字符级约束满足的词谜任务上的表现。跨家族差异产生了显著更大的性能差距(2.0-2.2倍,F1=0.761 vs. 0.343),而同一家族内参数扩展(从4B到32B)仅带来83%的提升。部分相关性分析排除了分词器设计作为同一家族内扩展的混淆因素。思考预算敏感性证明异质性:高容量模型表现出强回报(F1增加0.102至0.136),而中等规模模型饱和或退化,显示不一致的计算收益。利用10,000名人类求解者对每个谜题的难度评分,我们建立了所有家族中适度但一致的校准(ρ=0.28-0.42),但发现系统性失败在常见词上(如
英文摘要:
Large language models must satisfy hard orthographic constraints during controlled text generation, yet systematic cross-family evaluation remains limited. We evaluate 39 configurations spanning three model families (Qwen3, Claude Haiku 4.5, GPT-5-mini) on 58 word puzzles requiring character-level constraint satisfaction. Cross-family differences produce substantially larger performance gaps (2.0-2.2x, F1 = 0.761 vs. 0.343) than parameter scaling within families (83% gain from 4B to 32B scaling), and a partial-correlation analysis rules out tokenizer design as a confound for within-family scaling. Thinking budget sensitivity proves heterogeneous: high-capacity models show strong returns (+0.102 to +0.136 F1), while mid-sized variants saturate or degrade, showing inconsistent compute benefits. Using difficulty ratings from 10,000 human solvers per puzzle, we establish modest but consistent calibration (\r{ho} = 0.28-0.42) across all families, yet identify systematic failures on common words with unusual orthography ("data", "loll", "acai": 83-91% human success, 94-98% model miss rate). These failures point to over-reliance on distributional plausibility that penalizes orthographically atypical but constraint-valid patterns.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。