Evaluation and ranking of large language models (LLMs) has become an important problem with the proliferation of these models and their impact. Evaluation methods either require human responses which are expensive to acquire or use pairs of LLMs to evaluate each other which can be unreliable. In this paper, we provide a novel perspective where, given a dataset of prompts (viz. questions, instructions, etc.) and a set of LLMs, we rank them without access to any ground truth or reference responses. Inspired by real life where both an expert and a knowledgeable person can identify a novice our main idea is to consider triplets of models, where each one of them evaluates the other two, correctly identifying the worst model in the triplet with high probability. We also analyze our idea and provide sufficient conditions for it to succeed. Applying this idea repeatedly, we propose two methods to rank LLMs. In experiments on different generative tasks (summarization, multiple-choice, and dialog), our methods reliably recover close to true rankings without reference data. This points to a viable low-resource mechanism for practical use.
Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations
Seoul National University(首尔大学);NAVER AI Lab(NAVER人工智能实验室);NAVER CLOVA;Korea Advanced Institute of Science and Technology(韩国科学技术院);Hanyang University(汉阳大学)
Ground Truth Generation for Multilingual Historical NLP using LLMs
ARTFL Project, Romance Languages and Literatures, University of Chicago(ARTFL项目, Romance语言与文学系,芝加哥大学);Department of History, University of Chicago(历史系,芝加哥大学);Libraries and School of Information Studies, Purdue University(图书馆与信息科学学院,普渡大学)
Wisdom of the AI Crowd (AI-CROWD) for Ground Truth Approximation in Content Analysis: A Research Protocol & Validation Using Eleven Large Language Models
University of Alcalá(阿尔卡拉大学);Carlos III University of Madrid(马德里卡洛斯三世大学)
LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendation
Media Synthesis and Forensics Lab, Institute of Computing Technology, Chinese Academy of Sciences(媒体合成与取证实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variation
Independent Researcher(独立研究员);Amazon(亚马逊);Gazi University(加齐大学);University of Colorado(科罗拉多大学);Samsun University(萨姆松大学);The University Of Melbourne(墨尔本大学);New York University(纽约大学);RMIT University(皇家墨尔本理工大学);University of Pennsylvania(宾夕法尼亚大学)
Incident-Data Robustness Analysis of the OWASP Top 10 for LLM Applications (2026): How a Community-Expert Ranking Holds Up Against a Large-Scale LLM Incident Corpus
OWASP GenAI Security Project — Top 10 for LLM Applications(OWASP GenAI安全项目——LLM应用十大项目)
Beyond Single Ground Truth: Reference Monism as Epistemic Injustice in ASR Evaluation
Department of Information Science, Cornell University(康奈尔大学信息科学系);Department of Computer Science, Texas A&M University(德克萨斯农工大学计算机科学系);Rev AI;Tundra Technical Solutions(Tundra技术解决方案);Division of Liberal Studies, Kangwon National University(康sworth国立大学文科系)