Estimating model performance without labels is an important goal for understanding how NLP models generalize. While prior work has proposed measures based on dataset similarity or predicted correctness, it remains unclear when these estimates produce reliable performance rankings across domains. In this paper, we analyze the factors that affect ranking reliability using a two-step evaluation setup with four base classifiers and several large language models as error predictors. Experiments on the GeoOLID and Amazon Reviews datasets, spanning 15 domains, show that large language model-based error predictors produce stronger and more consistent rank correlations with true accuracy than drift-based or zero-shot baselines. Our analysis reveals two key findings: ranking is more reliable when performance differences across domains are larger, and when the error model's predictions align with the base model's true failure patterns. These results clarify when performance estimation methods can be trusted and provide guidance for their use in cross-domain model evaluation.
Modeling Relevance Ranking under the Pre-training and Fine-tuning Paradigm
School of Information, Renmin University of China(中国人民大学信息学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
Cross-institution text mining to uncover clinical associations: a case study relating social factors and code status in intensive care medicine
Bakar Computational Health Sciences Institute, University of California, San Francisco(加州大学旧金山分校巴卡尔计算健康科学研究所);Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University(乌得勒支大学医学中心尤利乌斯健康科学与初级保健中心)
IRLab, Informatics Institute, UvA, Amsterdam, the Netherlands(IR实验室、信息研究所、乌得勒支大学、阿姆斯特丹、荷兰);AI Lab, Pegasystems, Amsterdam, the Netherlands(AI实验室、佩加系统的阿姆斯特丹分部);LIACS, Leiden University, Leiden, the Netherlands(LIACS、莱顿大学、莱顿、荷兰)
Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers
Georgetown University(乔治城大学);University of Southern California(南加州大学);University of Maryland, College Park(马里兰大学学院公园分校);Arizona State University(亚利桑那州立大学)