SABER-Math：数学信息检索评估的自动化基准
SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
AI总结：
提出首个无需专家标注的数学信息检索自动化基准SABER-Math，通过三步构建重排序任务，评估检索器在数学领域的效果，发现通用基准无法可靠预测数学检索性能。
摘要：
随着智能体AI系统处理更复杂的数学任务，它们越来越依赖信息检索（IR）来搜索问题数据库、定理库和教育资源。然而，选择合适的检索器仍然困难，因为直接隔离其对下游性能的影响是不可行的。另一方面，现有的检索特定基准通常无法捕捉细粒度的数学相关性，从而惩罚相关文档。我们通过引入SABER-Math来解决这一差距，这是第一个无需专家标注的评估数学IR的完全自动化基准。从28.3万道高中水平数学题及其解答出发，SABER-Math通过三个步骤构建具有挑战性的重排序任务：（i）首先，LLM提取每个问题的简洁解答摘要和数学主题；（ii）然后，基于本体主题和词汇解答摘要相似性发现每个查询的相关文档；（iii）最后，采用瑞士风格的LLM偏好锦标赛为文档生成细粒度的相关性评级。我们评估了词汇检索器、专门的数学检索系统和最新的嵌入模型。我们发现，虽然现代嵌入模型显著优于经典和数学特定的基线，但即使是最强的系统在代数和微积分等符号密集的领域也表现挣扎。重要的是，我们表明通用IR基准（如MTEB）无法可靠预测数学性能，尤其是对于最新的嵌入模型，这凸显了对数学特定检索基准的需求。
英文摘要：
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to directly isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing relevant documents. We address this gap by introducing SABER-Math, the first fully automated benchmark for evaluating mathematical IR without expert annotation. Starting from 283K high-school-level math problems with solutions, SABER-Math builds challenging reranking tasks in three steps: (i) first, LLMs extract concise solution summaries and mathematical topics for each problem; (ii) then, per-query relevant documents are discovered using ontology topic-based and lexical solutions-summary-based similarities, and (iii) finally, a Swiss-style LLM preference tournament produces fine-grained relevance ratings for the documents. We evaluate lexical retrievers, specialized mathematical retrieval systems, and recent embedding models. We find that while modern embedding models substantially outperform classical and math-specific baselines, even the strongest systems struggle in symbol-heavy domains like Algebra and Calculus. Importantly, we show that general-purpose IR benchmarks such as MTEB do not reliably predict mathematical performance, especially for recent embedding models, highlighting the need for math-specific retrieval benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
