预期奖励预测,及其在模型路由中的应用
Expected Reward Prediction, with Applications to Model Routing
AI总结:
本文研究了基于响应级奖励模型预测模型对特定提示的适应性,提出了一种简单有效的预期奖励预测路由方法,实验证明其在模型路由中的优越性。
摘要:
本文研究了基于响应级奖励模型预测模型对特定提示的适应性,提出了一种简单有效的预期奖励预测路由方法,实验证明其在模型路由中的优越性。
英文摘要:
Reward models are a standard tool to score responses from LLMs. Reward models are built to rank responses to a fixed prompt sampled from a single model, for example to choose the best of n sampled responses. In this paper, we study whether scores from response-level reward models lifted to score a model's suitability for a prompt, prior to seeing responses from that model. Specifically, we show that it is straightforward to predict the expected reward that an LLM would earn from the reward model under repeated sampling. Further, we show that these expected reward predictions are precise and discriminative enough to support an application to a model routing protocol that routes prompts to models at inference time to maximize reward while controlling computational cost. We demonstrate the performance of this routing procedure on the open-perfectblend dataset, using a model pool composed of Llama3.1-Instruct 8B/70B, Gemma2-IT 9B/27B, and Gemma1-IT 7B models. Our simple expected reward prediction--based routing (ERP) outperforms baselines that route prompts to models with the best average performance within each prompt's category, and explains the success of more complex routing protocols that implicitly estimate an expected reward. Our approach has the added advantage of being trivially extensible as new models are added to the pool.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。