arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.09519cs.CL
收藏

我们能否在没有标注数据的情况下可靠地对跨领域模型性能进行排序?

Can We Reliably Rank Model Performance across Domains without Labeled Data?

The University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校);Illinois Institute of Technology(伊利诺伊理工学院)

Veronica Rammouz, Aaron Gonzalez, Carlos Cruzportillo, Adrian Tan, Nicole Beebe, Anthony Rios

AI总结:

本文研究无标签数据下跨领域模型性能排序的可靠性,通过两步评估实验发现基于大语言模型的错误预测器优于基线,且性能差异大、预测与真实失败模式一致时排名更可靠。

摘要:

在没有标签的情况下估计模型性能,是理解NLP模型如何泛化的重要目标。虽然先前的工作提出了基于数据集相似性或预测正确性的度量方法,但尚不清楚这些估计何时能在跨领域产生可靠的性能排名。在本文中,我们使用一个包含四个基础分类器和多个大型语言模型作为错误预测器的两步评估设置,分析了影响排名可靠性的因素。在涵盖15个领域的GeoOLID和Amazon Reviews数据集上的实验表明,基于大型语言模型的错误预测器产生的排名相关性与真实准确率相比,比基于漂移或零样本基线更强且更一致。我们的分析揭示了两个关键发现:当跨领域的性能差异更大时,排名更可靠;当错误模型的预测与基础模型的真实失败模式一致时,排名也更可靠。这些结果阐明了性能估计方法何时可以被信任,并为它们在跨领域模型评估中的使用提供了指导。

英文摘要:

Estimating model performance without labels is an important goal for understanding how NLP models generalize. While prior work has proposed measures based on dataset similarity or predicted correctness, it remains unclear when these estimates produce reliable performance rankings across domains. In this paper, we analyze the factors that affect ranking reliability using a two-step evaluation setup with four base classifiers and several large language models as error predictors. Experiments on the GeoOLID and Amazon Reviews datasets, spanning 15 domains, show that large language model-based error predictors produce stronger and more consistent rank correlations with true accuracy than drift-based or zero-shot baselines. Our analysis reveals two key findings: ranking is more reliable when performance differences across domains are larger, and when the error model's predictions align with the base model's true failure patterns. These results clarify when performance estimation methods can be trusted and provide guidance for their use in cross-domain model evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑