以千分之一成本调优LLM裁判的设计决策
Tuning LLM Judge Design Decisions for 1/1000 of the Cost
AI总结：
本文提出利用多目标多保真度方法系统调优LLM裁判的超参数，以千分之一成本找到兼顾准确性与成本效率的裁判，并采用开放权重模型提升可访问性与可复现性。
摘要：
评估大型语言模型（LLM）通常需要昂贵的人工标注。为解决这一问题，研究者提出了基于LLM的裁判（judge），通过比较两个LLM的输出，无需人工干预即可对模型进行排名。尽管已有多种方法被提出，但不同论文之间存在许多混淆因素。例如，模型、提示词及其他超参数通常同时改变，使得同类比较（apple-to-apple comparison）颇具挑战。本文提出系统性地分析与调优LLM裁判的超参数。为缓解评估裁判的高昂成本，我们提出利用多目标多保真度（multi-objective multi-fidelity）方法，该方法能够找到在准确性与成本之间权衡的裁判，同时显著降低搜索成本。我们的方法识别出的裁判不仅在准确性和成本效率上优于现有基准，而且采用开放权重模型，确保更高的可访问性和可复现性。复现实验的代码可在以下仓库获取：https://github.com/geoalgo/judgetuning。
英文摘要：
Evaluating Large Language Models (LLMs) often requires costly human annotations. To address this, LLM-based judges have been proposed, which compare the outputs of two LLMs enabling the ranking of models without human intervention. While several approaches have been proposed, many confounding factors are present between different papers. For instance the model, the prompt and other hyperparameters are typically changed at the same time making apple-to-apple comparisons challenging. In this paper, we propose to systematically analyze and tune the hyperparameters of LLM judges. To alleviate the high cost of evaluating a judge, we propose to leverage multi-objective multi-fidelity which allows to find judges that trade accuracy for cost and also significantly reduce the cost of the search. Our method identifies judges that not only outperform existing benchmarks in accuracy and cost-efficiency but also utilize open-weight models, ensuring greater accessibility and reproducibility. The code to reproduce our experiments is available at this repository https://github.com/geoalgo/judgetuning .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
