arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.15527cs.LGcs.CL
收藏

通过自适应采样实现大语言模型与文本生成图像模型的数据高效评估

Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling

Hewlett Packard Labs(惠普实验室)

Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Foltin Martin, Suparna Bhattacharya

AI总结:

针对大语言模型与文本生成图像模型评估计算成本高的问题,提出自适应采样框架SubLIME,通过多种采样策略构建基准代表性子集,在保证排名一致性的同时大幅降低评估成本。

摘要:

评估大语言模型(LLM)与文本生成图像模型是一项计算密集型任务,却常被忽视。高效评估对于理解这些模型的多样能力、以及在日益增多的新模型和基准之间开展对比至关重要。为解决这一问题,我们提出了SubLIME——一种数据高效的评估框架,它采用聚类、基于质量的方法等自适应采样技术来构建基准的代表性子集。我们的方法能确保模型排名与全数据集上的排名在统计上保持一致,这一点已通过较高的皮尔逊相关系数得到验证。在6个NLP基准上的实证分析表明:(1)在10%的采样率下,基于质量的采样(如Quality SE和Quality CPD)与全数据集的结果始终保持强相关性(0.85至0.95);(2)聚类方法在MMLU等特定基准上表现优异;(3)没有任何一种方法能在所有指标上普遍优于其他方法。对该框架进行扩展后,我们借助HEIM排行榜,覆盖了17个不同基准上的25个文本生成图像模型。SubLIME会为每个基准动态选择最优技术,在保留排名完整性和分数分布的同时大幅降低评估成本。值得注意的是,对于MMLU这类基准,低至1%的采样率就已足够有效。此外,我们证明采用基于难度的采样来针对更具挑战性的基准片段,能通过更宽泛的分数分布增强模型区分度。我们还结合语义搜索、工具使用和GPT-4评审,识别特定LLM类别(如编程基准)内各基准之间的冗余性,这让我们能进一步减少维持目标排名保留度所需的样本数量。总体而言,SubLIME为LLM和文本生成图像模型的稳健评估提供了一种通用且经济高效的解决方案。

英文摘要:

Evaluating LLMs and text-to-image models is a computationally intensive task often overlooked. Efficient evaluation is crucial for understanding the diverse capabilities of these models and enabling comparisons across a growing number of new models and benchmarks. To address this, we introduce SubLIME, a data-efficient evaluation framework that employs adaptive sampling techniques, such as clustering and quality-based methods, to create representative subsets of benchmarks. Our approach ensures statistically aligned model rankings compared to full datasets, evidenced by high Pearson correlation coefficients. Empirical analysis across six NLP benchmarks reveals that: (1) quality-based sampling consistently achieves strong correlations (0.85 to 0.95) with full datasets at a 10\% sampling rate such as Quality SE and Quality CPD (2) clustering methods excel in specific benchmarks such as MMLU (3) no single method universally outperforms others across all metrics. Extending this framework, we leverage the HEIM leaderboard to cover 25 text-to-image models on 17 different benchmarks. SubLIME dynamically selects the optimal technique for each benchmark, significantly reducing evaluation costs while preserving ranking integrity and score distribution. Notably, a minimal sampling rate of 1% proves effective for benchmarks like MMLU. Additionally, we demonstrate that employing difficulty-based sampling to target more challenging benchmark segments enhances model differentiation with broader score distributions. We also combine semantic search, tool use, and GPT-4 review to identify redundancy across benchmarks within specific LLM categories, such as coding benchmarks. This allows us to further reduce the number of samples needed to maintain targeted rank preservation. Overall, SubLIME offers a versatile and cost-effective solution for the robust evaluation of LLMs and text-to-image models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑