arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.20217cs.CLcs.LG
收藏

预期奖励预测,及其在模型路由中的应用

Expected Reward Prediction, with Applications to Model Routing

Stanford University Inception Labs(斯坦福大学Inception实验室);Stanford University Cognition Labs(斯坦福大学Cognition实验室);Google DeepMind(谷歌DeepMind);University of Chicago(芝加哥大学)

Kenan Hasanaliyev, Silas Alberti, Jenny Hamer, Dheeraj Rajagopal, Kevin Robinson, Jasper Snoek, Victor Veitch, Alexander Nicholas D'Amour

AI总结:

本文研究了基于响应级奖励模型预测模型对特定提示的适应性,提出了一种简单有效的预期奖励预测路由方法,实验证明其在模型路由中的优越性。

摘要:

本文研究了基于响应级奖励模型预测模型对特定提示的适应性,提出了一种简单有效的预期奖励预测路由方法,实验证明其在模型路由中的优越性。

英文摘要:

Reward models are a standard tool to score responses from LLMs. Reward models are built to rank responses to a fixed prompt sampled from a single model, for example to choose the best of n sampled responses. In this paper, we study whether scores from response-level reward models lifted to score a model's suitability for a prompt, prior to seeing responses from that model. Specifically, we show that it is straightforward to predict the expected reward that an LLM would earn from the reward model under repeated sampling. Further, we show that these expected reward predictions are precise and discriminative enough to support an application to a model routing protocol that routes prompts to models at inference time to maximize reward while controlling computational cost. We demonstrate the performance of this routing procedure on the open-perfectblend dataset, using a model pool composed of Llama3.1-Instruct 8B/70B, Gemma2-IT 9B/27B, and Gemma1-IT 7B models. Our simple expected reward prediction--based routing (ERP) outperforms baselines that route prompts to models with the best average performance within each prompt's category, and explains the success of more complex routing protocols that implicitly estimate an expected reward. Our approach has the added advantage of being trivially extensible as new models are added to the pool.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑