Large language models (LLMs) are increasingly used as automatic judges to evaluate system outputs in tasks such as reasoning, question answering, and creative writing. A faithful judge should base its verdicts solely on content quality, remain invariant to irrelevant context, and transparently reflect the factors driving its decisions. We test this ideal via controlled cue perturbations-synthetic metadata labels injected into evaluation prompts-for six judge models: GPT-4o, Gemini-2.0-Flash, Gemma-3-27B, Qwen3-235B, Claude-3-Haiku, and Llama3-70B. Experiments span two complementary datasets with distinct evaluation regimes: ELI5 (factual QA) and LitBench (open-ended creative writing). We study six cue families: source, temporal, age, gender, ethnicity, and educational status. Beyond measuring verdict shift rates (VSR), we introduce cue acknowledgment rate (CAR) to quantify whether judges explicitly reference the injected cues in their natural-language rationales. Across cues with strong behavioral effects-e.g., provenance hierarchies (Expert > Human > LLM > Unknown), recency preferences (New > Old), and educational-status favoritism-CAR is typically at or near zero, indicating that shortcut reliance is largely unreported even when it drives decisions. Crucially, CAR is also dataset-dependent: explicit cue recognition is more likely to surface in the factual ELI5 setting for some models and cues, but often collapses in the open-ended LitBench regime, where large verdict shifts can persist despite zero acknowledgment. The combination of substantial verdict sensitivity and limited cue acknowledgment reveals an explanation gap in LLM-as-judge pipelines, raising concerns about reliability of model-based evaluation in both research and deployment.
How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality
State Key Laboratory of AI Safety(人工智能安全国家重点实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Post and Telecommunications(北京邮电大学)
Truth or Twist? Optimal Model Selection for Reliable Label Flipping Evaluation in LLM-based Counterfactuals
Quality and Usability Lab, Technische Universität Berlin(柏林技术大学质量与可用性实验室);University of Marburg(马尔堡大学);German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心);Deutsche Telekom(德国电信);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究院)
Do LLM Self-Explanations Help Users Predict Model Behavior? Evaluating Counterfactual Simulatability with Pragmatic Perturbations
Faculty of Computer Science, University of Vienna(计算机科学系,维也纳大学);UniVie Doctoral School Computer Science, University of Vienna(UniVie 计算机科学博士学院,维也纳大学);Faculty of Philological and Cultural Studies, University of Vienna(文学与文化研究系,维也纳大学)
BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation
Southern University of Science and Technology(南方科技大学);Alibaba Group(阿里巴巴集团);Beihang University(北航);Shanghai University of Finance and Economics(上海财经大学)
SpatialChain: A Benchmark for Auditing Spatial Reasoning Faithfulness in VLMs
Universidade Federal de Mato Grosso (UFMT)(马托格罗索联邦大学);Universidade Federal de Goiás (UFG)(戈亚斯联邦大学);Advanced Knowledge Center for Immersive Technologies (AKCIT)(沉浸式技术高级知识中心)
Answer Matching Outperforms Multiple Choice for Language Model Evaluation
Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所);Tübingen AI Center(图宾根人工智能中心);University of Tübingen(图宾根大学)
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Yale University(耶鲁大学);University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校);VA Bedford Healthcare System(贝斯以色列女执事医疗中心贝德福德分部);Qingdao Medical College of Qingdao University(青岛大学青岛医学院);Yale School of Medicine(耶鲁医学院);National University Hospital, Singapore(新加坡国立大学医院);Zhejiang University(浙江大学);Stanford University(斯坦福大学)