arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07035cs.CL
收藏

统一的部署感知开放推理语言模型评估

Unified Deployment-Aware Evaluation of Open Reasoning Language Models

Department of Computer Science, Rensselaer Polytechnic Institute(理海理工学院计算机科学系);Department of Biomedical Engineering, Rensselaer Polytechnic Institute(理海理工学院生物医学工程系)

Md Motaleb Hossen Manik, Ge Wang

AI总结:

本文提出了一种统一的开放推理语言模型评估方法,通过四个基准测试(ARC-Challenge、GSM8K、MATH 1-3级和TruthfulQA MC1)对七种配置进行评估,结合零样本、链式思维(CoT)和少量样本CoT提示策略,分析模型在准确率、延迟、内存使用等多目标下的表现,强调部署感知的多目标优化问题。

摘要:

开放推理语言模型通常在混合样本量、部分标准化提示和以准确性为中心的总结下进行比较,这使得实际模型选择难以解释。我们针对ARC-Challenge、GSM8K、MATH 1-3级和TruthfulQA MC1四个基准测试,对七种开放推理语言模型配置进行了统一评估。我们对每种模型-数据集-策略条件下的238个示例子集测试了零样本、链式思维(CoT)和少量样本CoT提示策略,得到一个完整的7×4×3设计,包含84个条件和19,992个评估示例。除了准确性外,我们还报告了Wilson置信区间、延迟、峰值视频随机访问内存(VRAM)、加权聚合性能、帕累托高效运行点、提示敏感度指标和兼容性诊断。Gemma-4-26B-A4B在零样本提示下实现了最高的加权分数0.794。Gemma-4-E4B在各种提示设置中仍接近顶部,同时使用显著更低的延迟和内存,使其成为一种强大的实际运行点。Bootstrap和配对排列分析显示,领先配置足够接近,部署权衡仍然重要。我们还发现提示策略的变化会改变模型排名,而不是统一移动所有模型。基准特定的互补性创造了路由空间,一个 oracle 任务感知选择器达到了加权分数0.825。兼容性诊断显示,一些明显失败,尤其是Phi-4-Reasoning在GSM8K上的表现,反映了在共享评估流程下的鲁棒性和接口适应性问题。这些结果支持一个核心主张:开放模型评估应作为部署感知的多目标运行点问题,而不是单一分数排行榜练习。

英文摘要:

Open reasoning language models are often compared under mixed sample sizes, partially standardized prompts, and accuracy-centered summaries, which makes practical model selection difficult to interpret. We present a unified evaluation of seven open reasoning language model configurations across four benchmarks: ARC-Challenge, GSM8K, MATH levels 1 to 3, and TruthfulQA MC1. We test zero-shot, chain-of-thought (CoT), and few-shot CoT prompting on the same 238-example subset for every model--dataset--strategy condition, yielding a complete 7 x 4 x 3 design with 84 conditions and 19,992 evaluated examples. Beyond accuracy, we report Wilson confidence intervals, latency, peak video random access memory (VRAM), weighted aggregate performance, Pareto-efficient operating points, prompt-sensitivity metrics, and compatibility diagnostics. Gemma-4-26B-A4B with zero-shot prompting achieves the highest weighted score at 0.794. Gemma-4-E4B remains close to the top across prompting settings while using substantially lower latency and memory, making it a strong practical operating point. Bootstrap and paired-permutation analyses show that the leading configurations are close enough that deployment tradeoffs remain important. We also find that prompting strategy changes model rankings rather than shifting all models uniformly. Benchmark-specific complementarity creates routing headroom, with an oracle task-aware selector reaching a weighted score of 0.825. Compatibility diagnostics show that some apparent failures, especially Phi-4-Reasoning on GSM8K, reflect robustness and interface-adherence problems under the shared evaluation pipeline. These results support a central claim: open-model evaluation should be framed as a deployment-aware, multi-objective operating-point problem rather than as a single-score leaderboard exercise.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑