不要总是选择表现最好的模型：对大语言模型集成选择的信息论视角
Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection
AI总结：
本文提出了一种基于信息论的集成选择算法，通过最大化互信息来选择模型，以提高集成性能并避免性能饱和问题。
摘要：
大型语言模型（LLMs）通常被集成在一起以提高整体可靠性和鲁棒性，但在实践中模型之间存在很强的相关性。这提出了一个根本性问题：在形成LLM集成时应选择哪些模型？我们把预算有限的集成选择建模为最大化所选模型预测与真实标签之间的互信息。此外，为了解释为什么在许多模型的情况下性能可能饱和，我们使用高斯-协同模型来建模模型的相关误差，并展示了性能的信息论误差上限。受此启发，我们提出了一种简单的贪心互信息选择算法，该算法直接从数据中估计所需的信息项，并在查询预算下迭代构建集成。我们在两个问答数据集和一个二元情感分类数据集上测试了我们的方法：MEDMCQA、MMLU和IMDB电影评论。在所有数据集中，我们观察到我们的方法在相同查询预算下始终优于强大的基线方法。
英文摘要：
Large language models (LLMs) are often ensembled together to improve overall reliability and robustness, but in practice models are strongly correlated. This raises a fundamental question: which models should be selected when forming an LLM ensemble? We formulate budgeted ensemble selection as maximizing the mutual information between the true label and predictions of the selected models. Furthermore, to explain why performance can saturate even with many models, we model the correlated errors of the models using Gaussian-copula and show an information-theoretic error floor for the performance of the ensemble. Motivated by these, we propose a simple greedy mutual-information selection algorithm that estimates the required information terms directly from data and iteratively builds an ensemble under a query budget. We test our approach in two question answering datasets and one binary sentiment classification dataset: MEDMCQA, MMLU, and IMDB movie reviews. Across all datasets, we observe that our method consistently outperforms strong baselines under the same query budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
