arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.06172cs.AIcs.CL
收藏

加速语言模型评估

On Speeding Up Language Model Evaluation

Cornell University(康奈尔大学);University of California, San Diego(加利福尼亚大学圣迭戈分校)

Jin Peng Zhou, Christian K. Belardi, Ruihan Wu, Travis Zhang, Carla P. Gomes, Wen Sun, Kilian Q. Weinberger

AI总结:

本文提出一种自适应方法,利用多臂老虎机和低秩矩阵分解加速LLM提示方法评估,仅用5-15%资源即可识别最优方法,节省85-95%成本。

摘要:

使用大型语言模型(LLMs)开发基于提示的方法需要做出众多决策,这引发了超参数组合搜索问题。这种详尽评估可能耗时且昂贵。在本文中,我们提出了一种自适应方法来探索这一空间。我们利用了以下事实:通常只需少量样本即可识别出明显优越或较差的设置,并且许多评估测试高度相关。我们借助多臂老虎机算法顺序识别下一个要评估的(方法,验证样本)对,并利用低秩矩阵分解来填补缺失的评估。我们在多个竞争性基准问题上仔细评估了我们方法的有效性,并表明它仅使用典型资源的5-15%即可识别出表现最佳的方法——从而实现85-95%的LLM成本节省。我们的代码可在https://github.com/kilian-group/banditeval获取。

英文摘要:

Developing prompt-based methods with Large Language Models (LLMs) requires making numerous decisions, which give rise to a combinatorial search problem over hyper-parameters. This exhaustive evaluation can be time-consuming and costly. In this paper, we propose an $\textit{adaptive}$ approach to explore this space. We are exploiting the fact that often only few samples are needed to identify clearly superior or inferior settings, and that many evaluation tests are highly correlated. We lean on multi-armed bandits to sequentially identify the next (method, validation sample)-pair to evaluate and utilize low-rank matrix factorization to fill in missing evaluations. We carefully assess the efficacy of our approach on several competitive benchmark problems and show that it can identify the top-performing method using only 5-15% of the typical resources -- resulting in 85-95% LLM cost savings. Our code is available at https://github.com/kilian-group/banditeval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑