Assessing software test coverage at scale remains a bottleneck in QA pipelines. We present LLM-as-a-Judge (LAJ), a production-ready, rubric-driven framework for evaluating Gherkin acceptance tests with structured JSON outputs. Across 20 model configurations (GPT-4, GPT-5 with varying reasoning effort, and open-weight models) on 100 expert-annotated scripts over 5 runs (500 evaluations), we provide the first comprehensive analysis spanning accuracy, operational reliability, and cost. We introduce the Evaluation Completion Rate (ECR@1) to quantify first-attempt success, revealing reliability from 85.4% to 100.0% with material cost implications via retries. Results show that smaller models can outperform larger ones: GPT-4o Mini attains the best accuracy (6.07 MAAE), high reliability (96.6% ECR@1), and low cost ($1.01 per 1K), yielding a 78x cost reduction vs. GPT-5 (high reasoning) while improving accuracy. Reasoning effort is model-family dependent: GPT-5 benefits from increased reasoning (with predictable accuracy-cost tradeoffs), whereas open-weight models degrade across all dimensions as reasoning increases. Overall, cost spans 175x ($0.45-$78.96 per 1K). We release the dataset, framework, and code to support reproducibility and deployment.
LLM-as-Judge in Education: A Curriculum-Grounded Marking Pipeline
NSW Department of Education(新南威尔士州教育部);South Australian Department for Education(南澳大利亚州教育部);OC Selective exam preparation platform(OC精英考试备考平台);Studitory: HSC preparation platform(Studitory: HSC备考平台)
Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
AMAP, Alibaba Group(阿里巴巴集团AMAP实验室);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Southern California(南加州大学);University of Michigan, Ann Arbor(密歇根大学安娜堡分校)
JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
DataArc Tech Ltd.(DataArc科技有限公司);IDEA Research, International Digital Economy Academy(国际数字经济学院IDEA研究所);School of Advanced Interdisciplinary Sciences, UCAS(北京大学交叉学科学院);State Key Lab of AI Safety, Institute of Computing Technology, CAS(中国科学院计算技术研究所人工智能安全国家重点实验室)
Beyond Blind Spots: Analytic Hints for Mitigating LLM-Based Evaluation Pitfalls
Ora Nova Fandina(奥拉诺瓦·法丁纳);Eitan Farchi(埃itan·法奇);Shmulik Froimovich(什mulik·弗罗伊米奇);Raviv Gal(拉维夫·加尔);Wesam Ibraheem(韦萨姆·伊布拉希姆);Rami Katan(拉米·卡坦);Alice Podolsky(艾丽斯·波德洛斯基)
Judge Before Answer: Can MLLM Discern the False Premise in Question?
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);Inner Mongolia Research Institute, Shanghai Jiao Tong University(上海交通大学内蒙古研究院)
MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
School of Artificial Intelligence and Computer Science, Jilin University of Finance and Economics(吉林财经大学人工智能与计算机科学学院);Jilin Province Key Laboratory of Fintech, Jilin University of Finance and Economics(吉林省金融科技重点实验室,吉林财经大学);School of Engineering, Computer and Mathematical Sciences, Auckland University of Technology(奥克兰理工大学工程、计算机与数学科学学院)
UXBench: Measuring the Actionability of LLM-Generated UX Critiques
University of Notre Dame(诺丁汉大学);University of Pennsylvania(宾夕法尼亚大学);University of Rochester(罗切斯特大学);Carnegie Mellon University(卡内基梅隆大学);Massachusetts Institute of Technology(麻省理工学院);Harvard University(哈佛大学);LMU Munich(慕尼黑路德维希-马克西米利安大学)
Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions
Columbia University(哥伦比亚大学);The University of Texas at Austin(德克萨斯大学奥斯汀分校);The National Gallery of Art(国家艺术馆);UCLA(加州大学洛杉矶分校);UNC Chapel Hill(北卡罗来纳大学教堂山分校)