DualEval: 联合模型-项目校准的统一LLM评估
DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
AI总结：
提出DualEval框架，通过联合模型-项目校准统一静态基准和竞技场式评估，生成可靠模型排名并支持基准压缩和异常检测。
摘要：
当前的LLM评估依赖于两种互补但常常脱节的信号：具有客观正确性标签的静态基准和更能反映开放式用户交互的竞技场式偏好数据。我们引入DualEval，一个潜在模型-项目校准框架，将模型和评估项目表示在共享空间中，联合估计模型能力以及项目难度和锐度。我们在四个领域应用DualEval：编码、数学、杂项领域知识任务和通用日常用户查询。我们的评估使用了18个前沿LLM、静态基准标签以及针对保留的人类偏好验证的奖励模型分数，用于开放式模型响应。实验上，我们的框架产生可靠且平衡的模型排名，其学习的项目级概况支持下游应用，如用于样本高效评估的基准压缩和用于污染或异常检测的异常分析。总体而言，DualEval通过联合模型-项目校准统一了静态和竞技场式评估，产生模型排名和项目级诊断，支持更样本高效、可解释和可审计的评估流程。
英文摘要：
Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions. We introduce DualEval, a latent model-item calibration framework that represents models and evaluation items in a shared space, jointly estimating model ability together with item difficulty and sharpness. We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. Our evaluation uses 18 frontier LLMs, static benchmark labels, and reward-model scores validated against held-out human preferences for open-ended model responses. Empirically, our framework produces reliable and balanced model rankings, and its learned item-level profiles support downstream applications such as benchmark compression for sample-efficient evaluation and anomaly detection for contamination or outlier analysis. Overall, DualEval unifies static and arena-style evaluation through joint model-item calibration, producing model rankings and item-level diagnostics that support more sample-efficient, interpretable, and auditable evaluation pipelines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
