超越阿罗：多准则基准评估中的不可能性到可能性
Beyond Arrow: From Impossibility to Possibilities in Multi-Criteria Benchmarking
AI总结：
本文提出通过解决多准则基准评估中的不可能性问题，利用单峰、群体分离和距离受限偏好条件，实现稳定的模型排名。
摘要：
现代基准如HELM MMLU考虑了多个指标，如准确性、鲁棒性和效率。当试图将这些指标转化为单一排名时，自然的聚合过程可能会变得不一致或对模型集的变化不稳定。我们正式将这种聚合视为一个社会选择问题，其中每个指标在每个数据集上诱导对模型的偏好排名，而基准操作符在不同指标上聚合这些投票。尽管先前的工作集中在阿罗的不可能性结果上，我们认为这种不可能性往往源于病理例子，并识别出在哪些条件下这些例子消失，从而使得有意义的多准则基准评估成为可能。特别是，我们处理了排名组合的三个限制，并证明在单峰、群体分离和距离受限的偏好下，基准操作符允许构建良好的模型排名。经验上，我们研究了几个现代基准套件，如HELM MMLU，并验证了哪些结构性条件在哪些基准问题上得到满足。
英文摘要：
Modern benchmarks such as HELM MMLU account for multiple metrics like accuracy, robustness and efficiency. When trying to turn these metrics into a single ranking, natural aggregation procedures can become incoherent or unstable to changes in the model set. We formalize this aggregation as a social choice problem where each metric induces a preference ranking over models on each dataset, and a benchmark operator aggregates these votes across metrics. While prior work has focused on Arrow's impossibility result, we argue that the impossibility often originates from pathological examples and identify sufficient conditions under which these disappear, and meaningful multi-criteria benchmarking becomes possible. In particular, we deal with three restrictions on the combinations of rankings and prove that on single-peaked, group-separable and distance-restricted preferences, the benchmark operator allows for the construction of well-behaved rankings of the involved models. Empirically, we investigate several modern benchmark suites like HELM MMLU and verify which structural conditions are fulfilled on which benchmark problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
