重新思考文档排序中的推理：为什么思维链表现不佳
Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Short
AI总结：
本研究系统比较了有无思维链的重排序器，发现推理增强虽成本高但性能更差，直接微调更稳定有效，并指出了校准感知评分与简洁推理策略等改进方向。
摘要：
文档重排序是信息检索（IR）中的关键组成部分，旨在优化初始检索结果，以提高下游任务的排序质量。近期研究——受大型推理模型（LRMs）启发——已开始将显式思维链（CoT）推理融入基于LLM的重排序器中。然而，此类推理对排序任务的有效性仍未得到充分探索。在本工作中，我们首次系统研究了重排序中推理在逐点（pointwise）和列表式（listwise）两种设置下的表现，并覆盖了监督微调和强化学习两种训练方式。利用多样化的基准，包括推理密集型数据集（BRIGHT）和标准IR基准（BEIR），我们发现，尽管推理增强型重排序器产生了显著更高的推理成本，但其性能始终逊于直接预测排序且不使用CoT的对应模型。我们的分析揭示了三个核心局限：（i）在逐点重排序器中，推理破坏了校准，使模型偏向正类，提高了真阳性率（TPR）但降低了真阴性率（TNR），从而在负类占主导的候选池中增加了假阳性并损害了排序质量；（ii）在列表式重排序器中，推理改善了域内拟合，但增加了方差，且无法泛化到域外场景，即使强化学习缩短了推理链也依然如此；（iii）总体而言，直接微调的重排序器更加稳定、有效且鲁棒。这些发现挑战了显式推理对重排序普遍有益的假设。最后，我们指出了未来方向，包括为逐点重排序器设计校准感知的评分方法，以及设计简洁、有针对性的推理策略以缓解列表式重排序器中的过拟合和过度思考问题。
英文摘要：
Document reranking is a key component in information retrieval (IR), aimed at refining initial retrieval results to improve ranking quality for downstream tasks. Recent studies--motivated by large reasoning models (LRMs)--have begun incorporating explicit chain-of-thought (CoT) reasoning into LLM-based rerankers. However, the effectiveness of such reasoning for ranking tasks remains underexplored. In this work, we present the first systematic study of reasoning in reranking across both pointwise and listwise settings, under both supervised fine-tuning and reinforcement learning. Using diverse benchmarks, including reasoning-intensive datasets (BRIGHT) and standard IR benchmarks (BEIR), we find that reasoning-augmented rerankers consistently underperform their direct counterparts that predict rankings without CoT, despite substantially higher inference costs. Our analysis reveals three core limitations: (i) in pointwise rerankers, reasoning breaks calibration and biases models toward the positive class, raising TPR but lowering TNR, which inflates false positives and degrades ranking in negative-dominant pools; (ii) in listwise rerankers, reasoning improves in-domain fit but increases variance and fails to generalize out-of-domain, even when reinforcement learning shortens rationales; and (iii) overall, directly fine-tuned rerankers remain more stable, effective, and robust. These findings challenge the assumption that explicit reasoning is universally beneficial for reranking. We conclude by highlighting future directions, including calibration-aware scoring for pointwise rerankers and the design of concise, targeted reasoning strategies to mitigate overfitting and overthinking in listwise rerankers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
