arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.23293cs.GTcs.CY
收藏

聚合对模型多样性与消费者效用的影响

Impacts of Aggregation on Model Diversity and Consumer Utility

Massachusetts Institute of Technology(麻省理工学院);Siebel School of Computing and Data Science, University of Illinois, Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校西贝尔计算与数据科学学院);MIT Sloan School of Management(麻省理工学院斯隆管理学院);Department of Electrical Engineering and Computer Science(电气与计算机工程系)

Kate Donahue, Manish Raghavan

AI总结:

研究探讨了聚合对模型多样性和消费者效用的影响,提出加权胜率机制以提升生产者专业化和消费者福利。

摘要:

考虑一个AI工具的市场,每个工具都有略微不同的优势和劣势。通过为任务选择合适的模型,用户可以比单纯使用同一个模型更好。路由器在某种程度上遵循相似的原则,复杂的模型选择可以提高整体性能。然而,聚合往往嘈杂,反映了不完美的用户选择或路由决策。这导致两个主要问题:首先,什么样的“健康”模型市场能最大化消费者效用?其次,如何激励生产者创建这样的模型?我们证明,胜率,一个在LLM评估中的标准基准,可以激励模型创建者为了两种类型的模型变化而同质化,从而减少消费者福利。我们提出了一种新的机制,加权胜率,它奖励高质量的答案,并证明它能证明地提高生产者专业化和增加消费者福利。最后,我们通过经验基准数据集探讨了我们的理论结果的影响,并讨论了对基准设计的启示。

英文摘要:

Consider a marketplace of AI tools, each with slightly different strengths and weaknesses. By picking the right model for the task at hand, a user can do better than simply using the same model for everything. Routers operate under a similar principle, where sophisticated model selection can increase overall performance. However, aggregation is often noisy, reflecting imperfect user choices or routing decisions. This leads to two main questions: first, what does a "healthy marketplace" of models look like for maximizing consumer utility? Secondly, how can we incentivize producers to create such models? We show that winrate, a standard benchmark in LLM evaluation, can incentivize model creators to homogenize for both types of model changes, reducing consumer welfare. We propose a new mechanism, weighted winrate, which rewards models for answers that are higher quality, and show that it provably improves incentives for producers to specialize and increases consumer welfare. We conclude by exploring the impact of our theoretical results in empirical benchmark datasets and discussing implications for benchmark design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑