100个实例就足够:通过在少量实例上测试来预测新LLM在未见数据上的成功
100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances
AI总结:
本文提出利用少量参考实例和历史LLM评估结果训练通用评估器,以低成本预测新LLM在未见实例上的表现,并在HELM-Lite等基准上验证其分布内效果。
摘要:
预测LLM在单个任务实例上的性能,对于确保其在高风险应用中的可靠性至关重要。为此,一种可能的方法是在一组任务实例上评估所考虑的LLM,并训练一个评估器,根据实例的特征预测其性能。然而,这种方法需要在足够大的任务实例集上评估每个新LLM,才能训练出专用于该LLM的评估器。在本工作中,我们利用先前测试过的LLM的评估结果,以减少预测新LLM性能所需的评估次数。在实践中,我们建议在一小组参考实例上测试新LLM,并训练一个通用评估器,该评估器根据LLM在参考集上的性能以及目标实例的特征,预测其在该实例上的性能。我们在HELM-Lite和KindsOfReasoning上进行了实证研究;KindsOfReasoning是我们推出的一个现有推理数据集集合,在其中我们评估了截至2024年1月版GPT4之前的所有指令微调OpenAI模型。当在与用于训练通用评估器的实例具有相同分布的实例上预测性能时,我们发现该方法取得了与在完整实例集上训练的LLM专用评估器相当的性能。此外,我们发现随机选择参考实例的表现与我们测试的一些高级选择方法一样好。但在分布外情况下,没有明显的优胜者,且整体性能更差,这表明LLM的内在可预测性较低。
英文摘要:
Predicting the performance of LLMs on individual task instances is essential to ensure their reliability in high-stakes applications. To do so, a possibility is to evaluate the considered LLM on a set of task instances and train an assessor to predict its performance based on features of the instances. However, this approach requires evaluating each new LLM on a sufficiently large set of task instances to train an assessor specific to it. In this work, we leverage the evaluation results of previously tested LLMs to reduce the number of evaluations required to predict the performance of a new LLM. In practice, we propose to test the new LLM on a small set of reference instances and train a generic assessor which predicts the performance of the LLM on an instance based on the performance of the former on the reference set and features of the instance of interest. We conduct empirical studies on HELM-Lite and KindsOfReasoning, a collection of existing reasoning datasets that we introduce, where we evaluate all instruction-fine-tuned OpenAI models until the January 2024 version of GPT4. When predicting performance on instances with the same distribution as those used to train the generic assessor, we find this achieves performance comparable to the LLM-specific assessors trained on the full set of instances. Additionally, we find that randomly selecting the reference instances performs as well as some advanced selection methods we tested. For out of distribution, however, no clear winner emerges and the overall performance is worse, suggesting that the inherent predictability of LLMs is low.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。