Evaluation of text generated by large language models (LLMs) is increasingly delegated to LLMs acting as judges, yet what such a judge must be shown remains unresolved. We treat a judge whose reports are acted on as a regulator, in the sense of cybernetics, and ask what it must be shown, how much, in what form, and what its reports must get right. On seven public benchmarks we varied only what the judge was shown. Across three judges whose prices differ 10- to 17-fold, what the judge was shown mattered more than which judge it was. When a system revised its answers on feedback from a judge denied the material they had to respect, it ended below its starting point on instruction following and question answering. On instruction following the loss grew with each round, and it persisted when the judge saw the user's request but not the constraints and with a second judge family. On news summarisation, a correct reference summary beside the source undid most of the source's contribution, whereas on single-answer questions a reference helped. About a tenth of the source did almost all of its work, and a small entailment graph matched an LLM judge at finding errors in grounded text. We committed 83 testable clauses before running; of the 81 that ran, 32 met their bar and 50 went in the predicted direction, and most quantitative predictions from the theory failed. We report every shortfall and derive design rules tied to measured effects.
Blinded Radiologist and LLM-Based Evaluation of LLM-Generated Japanese Translations of Chest CT Reports: Comparative Study
Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo(东京大学医学研究生院放射学与生物医学工程系);Department of Radiology, The University of Tokyo Hospital(东京大学医院放射科);Department of Radiology, Kanazawa University Graduate School of Medical Sciences(金泽大学医学研究生院放射科);Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital(东京大学医院计算诊断放射学与预防医学系);Department of Radiology, School of Medicine, Jichi Medical University(自治医科大学医学院放射科)
Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls
Ora Nova Fandina(奥拉诺瓦·法丁纳);Eitan Farchi(埃itan·法奇);Shmulik Froimovich(什mulik·弗罗伊米奇);Raviv Gal(拉维夫·加尔);Wesam Ibraheem(韦萨姆·伊布拉希姆);Rami Katan(拉米·卡坦);Alice Podolsky(艾丽斯·波德洛斯基)
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
Technical University of Munich (TUM)(慕尼黑技术大学);Ludwig Maximilian University of Munich (LMU)(慕尼黑路德维希-马克西米利安大学);University of Konstanz(康斯坦茨大学);University of Saarbrücken(萨尔布吕肯大学)
Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
School of Computing and Electrical Engineering(计算与电子工程学院);Indian Institute of Technology (IIT) Mandi(印度理工学院(IIT)曼迪);London U.K.(伦敦英国);Warwick Manufacturing Group U.K.(沃里克制造集团英国)
Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judge
School of Computing and Augmented Intelligence, Arizona State University(亚利桑那州立大学计算与增强智能学院);School of Mathematical and Natural Sciences, Arizona State University(亚利桑那州立大学数学与自然科学学院)
A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models
Technical University of Munich(慕尼黑技术大学);Foundation for Research and Technology -- Hellas(希腊研究与技术基金会);National University of Science and Technology Politehnica Bucharest(布加勒斯特政治技术科学与技术国家大学)
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Department of Biomedical Informatics, School of Medicine, University of Pittsburgh(匹兹堡大学医学院生物医学信息学系);Department of Health Information Management, School of Health and Rehabilitation Sciences, University of Pittsburgh(匹兹堡大学健康与康复科学学院健康信息管理系);OpenCura, Health Innovation Consortium(OpenCura健康创新联盟);Northwestern University, Kellogg School of Management(西北大学凯洛格管理学院);Intelligent Systems Program, School of Computing and Information, University of Pittsburgh(匹兹堡大学计算与信息学院智能系统项目);Johns Hopkins University School of Medicine Biomedical Informatics and Data Science(约翰霍普金斯大学医学院生物医学信息学与数据科学);Clinical and Translational Science Institute, University of Pittsburgh(匹兹堡大学临床与转化科学研究所);Institute for Biomedical Informatics, University of Pennsylvania(宾夕法尼亚大学生物医学信息学研究所);Data Science, School of Computing and Information, University of Pittsburgh(匹兹堡大学计算与信息学院数据科学)
UXBench: Measuring the Actionability of LLM-Generated UX Critiques
University of Notre Dame(诺丁汉大学);University of Pennsylvania(宾夕法尼亚大学);University of Rochester(罗切斯特大学);Carnegie Mellon University(卡内基梅隆大学);Massachusetts Institute of Technology(麻省理工学院);Harvard University(哈佛大学);LMU Munich(慕尼黑路德维希-马克西米利安大学)