arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02515cs.AIcs.CLcs.LG
收藏

CreditAudit: 2维度用于LLM评估与选择

CreditAudit: 2$^\text{nd}$ Dimension for LLM Evaluation and Selection

Northwestern Polytechnical University Institute of Artificial Intelligence (TeleAI), China Telecom(西北工业大学人工智能研究所(TeleAI),中国电信);Dalian Maritime University Institute of Artificial Intelligence (TeleAI), China Telecom(大连海事大学人工智能研究所(TeleAI),中国电信);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究所(TeleAI),中国电信);Guangxi Normal University Institute of Artificial Intelligence (TeleAI), China Telecom(广西师范大学人工智能研究所(TeleAI),中国电信)

Yiliang Song, Hongjun An, Jiangong Xiao, Haofei Zhao, Jiawei Shao, Xuelong Li

AI总结:

CreditAudit 通过 2D 信用评级框架,提供基于稳定性风险的 LLM 部署评估方法,支持更精确的模型选择与分层部署。

摘要:

公开基准 leaderboard 上的分数一直在稳步上升并趋于收敛,许多前沿语言模型现在仅以微小差异分隔开。然而,这些分数往往无法匹配用户日常体验,因为系统提示、输出协议和交互模式在常规迭代中不断演变,在代理多步骤流程中,小的协议变化可能引发不成比例的失败,使从业者不确定应部署哪个模型。我们提出了 CreditAudit,一个面向部署的信用审计框架,该框架在多个基准上,使用一组语义对齐且非对抗性的系统提示模板评估模型,报告平均能力作为在不同场景中的平均性能,以及场景诱导的波动性 sigma 作为稳定性风险信号,并进一步通过跨模型分位数将波动性映射到可解释的信用等级(从 AAA 到 BBB),通过诊断来缓解模板难度漂移。在 GPQA、TruthfulQA 和 MMLU Pro 上的受控实验表明,具有相似平均能力的模型可以表现出显著不同的波动性,并且稳定性风险可以在代理或高失败成本的 regime 中推翻优先级决策。通过提供基于 2D 和等级的语言进行特定 regime 的选择,CreditAudit 支持分层部署,并更合理地分配测试和监控努力,使模型在现实应用中的评估更加客观和可信。

英文摘要:

Leaderboard scores on public benchmarks have been steadily rising and converging, with many frontier language models now separated by only marginal differences. However, these scores often fail to match users' day to day experience, because system prompts, output protocols, and interaction modes evolve under routine iteration, and in agentic multi step pipelines small protocol shifts can trigger disproportionate failures, leaving practitioners uncertain about which model to deploy. We propose CreditAudit, a deployment oriented credit audit framework that evaluates models under a family of semantically aligned and non adversarial system prompt templates across multiple benchmarks, reporting mean ability as average performance across scenarios and scenario induced fluctuation sigma as a stability risk signal, and further mapping volatility into interpretable credit grades from AAA to BBB via cross model quantiles with diagnostics that mitigate template difficulty drift. Controlled experiments on GPQA, TruthfulQA, and MMLU Pro show that models with similar mean ability can exhibit substantially different fluctuation, and stability risk can overturn prioritization decisions in agentic or high failure cost regimes. By providing a 2D and grade based language for regime specific selection, CreditAudit supports tiered deployment and more disciplined allocation of testing and monitoring effort, enabling more objective and trustworthy model evaluation for real world use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑