While Large Language Models (LLMs) are increasingly adopted as automated judges for evaluating generated text, their outputs are often costly, and highly sensitive to prompt design, language, and aggregation strategies, severely, which limits reproducibility. To address these challenges, we propose \textbf{\textit{OmniScore}}, a family of complementary, deterministic learned metrics developed using small size ($<$1B) parameter models. OmniScore approximates LLM-judge behavior while preserving the low latency and consistency of traditional model-based scoring. We trained the models large-scale synthetic supervision ($\sim$564k instances, in \textbf{107 languages}) and evaluated using 8,617 manually annotated instances. The OmniScore family supports reliable, multi-dimensional scores across a variety of settings, including reference-based, source-grounded, and hybrid evaluations. We evaluate these models across question answering (QA), translation, and summarization in \textbf{6 languages}. Our results demonstrate that lightweight, deterministic learned metrics provide a highly practical and scalable alternative to frontier LLMs. Our models and datasets can be found at https://huggingface.co/collections/QCRI/omniscore
Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages
LT3, IDLab, Universiteit Gent, Barcelona Supercomputing Center, LMU Munich & Munich Center for Machine Learning, German Center for Addiction Research in Childhood and Adolescence, University Medical Center Hamburg-Eppendorf, Mila - Quebec AI Institute, McGill University, Canada CIFAR AI Chair(LT3、IDLab、根特大学、巴塞罗那超级计算中心、慕尼黑莱茵河大学及慕尼黑机器学习中心、德国成年期成瘾研究中心、汉堡埃彭多夫大学医学中心、魁北克人工智能研究所、麦吉尔大学、加拿大 CIFAR 人工智能主席)
MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
Yonsei University(延世大学);KAIST(韩国科学技术院);OneLineAI;Barcelona Supercomputing Center(巴塞罗那超级计算中心);Artful Media;Bangladesh University of Engineering and Technology(孟加拉国工程技术大学);Carnegie Mellon University(卡内基梅隆大学)
Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond English
HiTZ Center, University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU)HiTZ中心);University of Alicante (UA)(阿利坎特大学(UA));GOI Institute, Basque Summer University (UEU)(巴斯克夏季大学(UEU)GOI研究所)
WorldBench: Culturally Grounded Benchmark for Multilingual Agents
University of Edinburgh(爱丁堡大学);School of Informatics, University of Edinburgh(爱丁堡大学信息学院);School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院)
IKS-Instruct: A 24,000-Example Multilingual Dataset for Teaching Language Models Indian Knowledge Systems
Intrinsic Lab, Centre for Sensors, Instrumentation and Cyber-Physical System Engineering (Centre for SeNSE), Indian Institute of Technology Delhi(印度理工学院德里分校传感器、仪器仪表和网络物理系统工程中心(传感中心)本征实验室);RSL Quantum, FITT, IIT Delhi(印度理工学院德里分校 FITT 学院 RSL 量子实验室)
GRASP: Deterministic argument ranking in interaction graphs
MPI-IS Tübingen(图宾根MPI研究所);Tübingen AI Center(图宾根人工智能中心);ELLIS Institute Tübingen(图宾根ELLIS研究所);Eberhard Karls Universität Tübingen(图宾根埃伯哈德·卡尔斯大学);LIONS, EPFL(EPFL的LIONS实验室)
A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models
Technical University of Munich(慕尼黑技术大学);Foundation for Research and Technology -- Hellas(希腊研究与技术基金会);National University of Science and Technology Politehnica Bucharest(布加勒斯特政治技术科学与技术国家大学)
Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi
Bonn-Aachen International Center for Information Technology (b-it) / CAISA Lab(波恩-亚琛国际信息科技中心(b-it)/ CAISA实验室);Lamarr Institute for Machine Learning and Artificial Intelligence(拉玛尔人工智能与机器学习研究所);Bonn Sustainable AI Lab(波恩可持续AI实验室);Helmholtz-Institut für Strahlen- und Kernphysik(海德堡辐射与核物理研究所)