RAR-b：Reasoning as Retrieval Benchmark（将推理作为检索的基准）
RAR-b: Reasoning as Retrieval Benchmark
AI总结：
本文提出 RAR-b 基准，将推理任务转化为检索任务以评估检索器的推理能力，发现现有模型存在明显短板，并通过微调重排序器在所有任务上达到最优。
摘要：
在过去几年中，语义文本相似度（STS）和信息检索（IR）任务一直是记录嵌入模型进展的两大主要途径。在新兴的检索增强生成（RAG）范式下，我们设想有必要评估嵌入模型的下一层级语言理解能力，并有意识地审视其中存储的推理能力。为解决这一问题，我们提出疑问：检索器能否解决推理问题？通过将推理任务转化为检索任务，我们发现，如果没有针对推理层级的语言理解进行专门训练，当前最先进的检索器模型可能仍远不足以胜任辅助 LLM 的角色，尤其是在推理密集型任务中。此外，尽管指令感知型 IR 模型经过训练以理解指令，但在推理任务的推理阶段，不使用指令往往效果更好，这暴露出一个被忽视的检索器—LLM 行为差距，有待研究界加以对齐。然而，近期基于解码器的嵌入模型在缩小这一差距方面展现出巨大潜力，突显了嵌入模型实现推理层级语言理解的路径。我们还表明，尽管当前现成的重排序器模型在这些任务上表现失败，但通过微调向其注入推理能力，似乎仍比对双编码器这样做更容易；而且通过微调一个重排序模型，我们能够在所有任务上取得最先进性能。我们发布了 Reasoning as Retrieval Benchmark（RAR-b），这是一套用于评估检索器模型所存储推理能力的整体性任务与设置套件。RAR-b 可在 https://github.com/gowitheflow-1998/RAR-b 获取。
英文摘要：
Semantic textual similartiy (STS) and information retrieval tasks (IR) tasks have been the two major avenues to record the progress of embedding models in the past few years. Under the emerging Retrieval-augmented Generation (RAG) paradigm, we envision the need to evaluate next-level language understanding abilities of embedding models, and take a conscious look at the reasoning abilities stored in them. Addressing this, we pose the question: Can retrievers solve reasoning problems? By transforming reasoning tasks into retrieval tasks, we find that without specifically trained for reasoning-level language understanding, current state-of-the-art retriever models may still be far from being competent for playing the role of assisting LLMs, especially in reasoning-intensive tasks. Moreover, albeit trained to be aware of instructions, instruction-aware IR models are often better off without instructions in inference time for reasoning tasks, posing an overlooked retriever-LLM behavioral gap for the research community to align. However, recent decoder-based embedding models show great promise in narrowing the gap, highlighting the pathway for embedding models to achieve reasoning-level language understanding. We also show that, although current off-the-shelf re-ranker models fail on these tasks, injecting reasoning abilities into them through fine-tuning still appears easier than doing so to bi-encoders, and we are able to achieve state-of-the-art performance across all tasks by fine-tuning a reranking model. We release Reasoning as Retrieval Benchmark (RAR-b), a holistic suite of tasks and settings to evaluate the reasoning abilities stored in retriever models. RAR-b is available at https://github.com/gowitheflow-1998/RAR-b.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
