Shanghai Jiao Tong University(上海交通大学);Shanghai AI Lab(上海人工智能实验室);Tsimnghua University(塔密纳大学);Beijing University of Posts and Telecommunications(北京邮电大学);Zhejiang University(浙江大学);University of Tokyo(东京大学)
Long-Form Question Answering (LFQA) involves generating comprehensive, paragraph-level responses to open-ended questions, which poses a significant challenge for evaluation due to the richness of information and flexible response format. Existing LFQA-evaluation benchmarks often lack reference answers and are limited in size and topic coverage, reducing their reliability. To address this gap, we introduce LFQA-E, a well-constructed, multilingual, and reference-based benchmark designed to rigorously evaluate automatic metrics for LFQA. LFQA-E comprises 1618 questions and 7323 pairwise comparisons across 15 topics, drawn from diverse sources such as online queries and examination questions, thereby enabling a comprehensive assessment of evaluation metrics. We examine five categories of metrics, encompassing 17 specific methods, using LFQA-E. The results demonstrate that none of the existing automatic metrics perform comparably to human judgments, highlighting their inability to capture the dense information in long-form responses. Furthermore, we present a detailed analysis of the failure cases and the generalization capacity of these metrics, offering insights to guide the future development of LFQA evaluation methods. The benchmark and code are available at https://github.com/YuchenFan48/LFQA-E.
KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Hong Kong University of Science and Technology(香港科技大学);The University of Osaka(大阪大学)
Beyond Isolated Dots: Benchmarking Structured Table Construction as Deep Knowledge Extraction
University of Chinese Academy of Sciences(中国科学院大学);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室)
RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);Alibaba Group(阿里巴巴集团);University of Chinese Academy of Sciences(中国科学院大学)
Performance of large language models in numerical vs. semantic medical knowledge: Benchmarking on evidence-based Q&As
Tel-Aviv University(特拉维夫大学);Kahun Medical Ltd(Kahun医疗有限公司);Hebrew University of Jerusalem(耶路撒冷希伯来大学);Leumit Health Services(Leumit医疗服务机构);Bar-Ilan University(巴伊兰大学);Kaplan Medical Center(卡普兰医疗中心);Brigham and Women’s Hospital(布莱根妇女医院);Harvard Medical School(哈佛医学院)
From Words to Worlds: Benchmarking Cross-Cultural Cultural Understanding in Machine Translation
Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences Urumqi China;Xinjiang Technical Institute of Physics \& Chemistry, Chinese Academy of Sciences
InnoGym: Benchmarking the Innovation Potential of AI Agents
Zhejiang University(浙江大学);Ant Group(蚂蚁集团);National University of Singapore(新加坡国立大学);Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph(知识图谱联合实验室)
HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models
Central South University(中南大学);Tsinghua University(清华大学);South China Normal University(华南师范大学);ByteDance Inc(字节跳动公司);University of Zaragoza(阿拉维达大学);CosmosMind;Wuhan University(武汉大学);University of California, Los Angeles(加州大学洛杉矶分校);Southeast University(东南大学);Tencent(腾讯公司);Nankai University(南开大学);Supermicro Computer Inc(Supermicro计算机公司);Huazhong University of Science and Technology(华中科技大学)
CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks
Alibaba Group(阿里巴巴集团);Beijing Language and Culture University(北京语言大学);LMU Munich(慕尼黑大学);ELLIS Institute Finland(芬兰ELLIS研究所);University of Turku(图尔库大学);IBM Research AI, UAE(阿联酋IBM人工智能研究);MBZUAI;Harbin Institute of Technology, Shenzhen(深圳哈尔滨工业大学);Southern University of Science and Technology(南方科技大学)