Large language models (LLMs) are increasingly used to support question answering and decision-making in high-stakes, domain-specific settings such as natural hazard response and infrastructure planning, where effective answers must convey fine-grained, decision-critical details. However, existing evaluation frameworks for retrieval-augmented generation (RAG) and open-ended question answering primarily rely on surface-level similarity, factual consistency, or semantic relevance, and often fail to assess whether responses provide the specific information required for domain-sensitive decisions. To address this gap, we propose a multi-dimensional, reference-free evaluation framework that assesses LLM outputs along four complementary dimensions: specificity, robustness to paraphrasing and semantic perturbations, answer relevance, and context utilization. We introduce a curated dataset of 1,412 domain-specific question-answer pairs spanning 40 professional roles and seven natural hazard types to support systematic evaluation. We further conduct human evaluation to assess inter-annotator agreement and alignment between model outputs and human judgments, which highlights the inherent subjectivity of open-ended, domain-specific evaluation. Our results show that no single metric sufficiently captures answer quality in isolation and demonstrate the need for structured, multi-metric evaluation frameworks when deploying LLMs in high-stakes applications.
Sentinel: Decoding Context Utilization via Attention Probing for Efficient LLM Context Compression
Ping An Technology (Shenzhen) Co., Ltd., China(平安科技(深圳)有限公司,中国);University of Science and Technology of China(中国科学技术大学);University of Electronic Science and Technology of China(电子科技大学)
REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);National University of Singapore(新加坡国立大学);Baidu Inc.(百度公司);School of Information, Renmin University of China(中国人民大学信息学院)
BELIEF: Structured Evidence Modeling and Uncertainty-Aware Fusion for Biomedical Question Answering
School of Computer Science and Technology, Zhejiang University of Science and Technology(浙江理工大学计算机科学与技术学院);College of Artificial Intelligence, Zhejiang University(浙江大学人工智能学院);Zhejiang Key Laboratory of Biomedical Intelligent Computing Technology, Zhejiang University of Science and Technology(浙江理工大学生物医学智能计算技术重点实验室)
EMPOWER: Evolutionary Medical Prompt Optimization With Reinforcement Learning
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(脑启发智能感知与认知国家重点实验室,中国科学技术大学);Department of Computer Science, University of Minnesota-Twin Cities(计算机科学系,明尼苏达大学双城分校);Center of Research for Cyber Security and Network (CSNET), Faculty of Computer Science and Information Technology, Universiti Malaya(网络安全与网络研究中心(CSNET),马来亚大学计算机科学与信息技术学院);DSLAB, School of Information Science & Engineering, Lanzhou University(信息科学与工程学院,兰州大学);Institute of Computing Technology, Chinese Academy of Sciences(计算技术研究所,中国科学院);Department of AI, Prince Mohammad bin Fahd University(人工智能系,普林姆·法赫德大学);Department of Computer Engineering, College of Computer Science, King Khalid University(计算机工程系,国王·卡利德大学)
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges
Peking University(北京大学);Institute of Information Engineering, CAS(中国科学院信息工程研究所);AAII, FEIT, University of Technology Sydney(悉尼科技大学);UCLA(加利福尼亚大学洛杉矶分校);The Open University of China(国家开放大学);Beihang University(北京航空航天大学)