Think-Probe-Respond：提升大型语言模型作为研究创意新颖性评判者的能力
Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty
浏览论文内容
中文总结 AI 辅助
该研究针对大型语言模型评判研究创意新颖性时的“中等新颖”偏差，提出TPR轻量方法，通过探测推理阶段隐藏状态的潜在评判约束最终响应，使性能提升22.30%。
中文摘要 AI 辅助
自动化新颖性评判可通过高效评估、优化和对比研究创意来加速科学发现。尽管大型语言模型越来越多地被用于该任务，但我们研究了其评判能力中一个此前被忽视的局限：尽管生成的推理依据与人类专家的高度相似，其最终新颖性评判却常出现显著偏差。我们表明这种校准错误源于一种系统性偏向，即倾向于将创意评判为“中等新颖”。为缓解此问题，我们提出Think-Probe-Respond（TPR），一种轻量方法，该方法在推理阶段从隐藏状态中探测潜在的新颖性评判，并使用探测到的评判来约束最终响应。在多个强基准上，TPR将新颖性评判性能提升了22.30%，并成功缓解了普遍存在的“中等新颖”偏差。
英文摘要
Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR), a lightweight approach that probes latent novelty judgments from hidden states during the reasoning phase and uses the probed judgments to condition the final response. Across strong baselines, TPR improves novelty judgment performance by 22.30% and successfully mitigates the prevalent "medium novelty" bias.
发表机构
- National Institute of Informatics（情报研究综合研究所）
- TU Dresden（德累斯顿工业大学）
- ScaDS.AI Dresden/Leipzig（德累斯顿/莱比锡应用数据科学中心）
机构由 AI 辅助整理，请以论文原文为准。
