语言大模型 / LLM

HealthQA-BR: A System-Wide Benchmark Reveals Critical Knowledge Gaps in Large Language Models

Andrew Maranhão Ventura D'addario

机构 * Independent Researcher（独立研究者） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency

Shiji Zhao, Ranjie Duan, Fengxiang Wang, Chi Chen, Caixin Kang, Shouwei Ruan, Jialing Tao, YueFeng Chen, Hui Xue, Xingxing Wei

机构 * Institution1（机构1） ； Institution2（机构2） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments ICCV2025

EUR-USD Exchange Rate Forecasting Based on Information Fusion with Large Language Models and Deep Learning Methods

Hongcheng Ding, Xuanze Zhao, Ruiting Deng, Shamsul Nahar Abdullah, Deshinta Arrova Dewi

机构 * INTI International University（INTI国际大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages

Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Liu, Haobin Lin, Bingni Zhang, Taifeng Wang, Mykola Pechenizkiy, Meng Fang, Yin Zheng

机构 * Eindhoven University of Technology（埃因霍温理工大学） ； ByteDance（字节跳动） ； University of Liverpool（利物浦大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

The Democratic Paradox in Large Language Models' Underestimation of Press Freedom

I. Loaiza, R. Vestrelli, A. Fronzetti Colladon, R. Rigobon

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Batayan: A Filipino NLP benchmark for evaluating Large Language Models

Jann Railey Montalan, Jimson Paulo Layacan, David Demitri Africa, Richell Isaiah Flores, Michael T. Lopez, Theresa Denise Magsajo, Anjanette Cayabyab, William Chandra Tjhi

机构 * AI Singapore（AI新加坡） ； National University of Singapore（新加坡国立大学） ； Ateneo de Manila University（马尼拉大学） ； University of Cambridge（剑桥大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to ACL 2025 (Main Conference)

When language and vision meet road safety: leveraging multimodal large language models for video-based traffic accident analysis

Ruixuan Zhang, Beichen Wang, Juexiao Zhang, Zilin Bian, Chen Feng, Kaan Ozbay

机构 * New York University（纽约大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Large Language Models as 'Hidden Persuaders': Fake Product Reviews are Indistinguishable to Humans and Machines

Weiyao Meng, John Harvey, James Goulding, Chris James Carter, Evgeniya Lukinova, Andrew Smith, Paul Frobisher, Mina Forrest, Georgiana Nica-Avram

机构 * N/LAB, Nottingham University Business School, University of Nottingham, UK（N/LAB，诺丁汉大学商学院，诺丁汉大学，英国） ； Haydn Green Institute for Entrepreneurship and Innovation, University of Nottingham, UK（创业与创新研究院，诺丁汉大学，英国） ； Strategic Innovation Ltd, UK（战略创新有限公司，英国） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

RealFactBench: A Benchmark for Evaluating Large Language Models in Real-World Fact-Checking

Shuo Yang, Yuqin Dai, Guoqing Wang, Xinran Zheng, Jinfeng Xu, Jinze Li, Zhenzhe Ying, Weiqiang Wang, Edith C. H. Ngai

机构 * The University of Hong Kong（香港大学） ； Tsinghua University（清华大学） ； University College London（伦敦大学学院） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

OR-Bench: An Over-Refusal Benchmark for Large Language Models

Justin Cui, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICML 2025, we thank everyone for their valuable suggestions and feedback!

KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations

Junyu Liu, Kaiqi Yan, Tianyang Wang, Qian Niu, Momoko Nagai-Tanima, Tomoki Aoyama

机构 * Kyoto University（京都大学） ； The Hong Kong University of Science and Technology（香港科技大学） ； Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学） ； The University of Tokyo（东京大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments 9pages, 3 figures

TeleEval-OS: Performance evaluations of large language models for operations scheduling

Yanyan Wang, Yingying Wang, Junli Liang, Yin Xu, Yunlong Liu, Yiming Xu, Zhengwang Jiang, Zhehe Li, Fei Li, Long Zhao, Kuang Xu, Qi Song, Xiangyang Li

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models

Bang An, Sicheng Zhu, Ruiyi Zhang, Michael-Andrei Panaitescu-Liess, Yuancheng Xu, Furong Huang

机构 * University of Maryland, College Park（马里兰大学 College Park 分校） ； Adobe Research（Adobe 研究院） ； Capital One（Capital One 公司） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.LG

Epistemic Integrity in Large Language Models

Bijean Ghafouri, Shahrad Mohammadzadeh, James Zhou, Pratheeksha Nair, Jacob-Junqi Tian, Hikaru Tsujimura, Mayank Goel, Sukanya Krishna, Reihaneh Rabbany, Jean-François Godbout, Kellin Pelrine

机构 * University of Southern California（南加州大学） ； McGill University（麦吉尔大学） ； UC Berkeley（加州大学伯克利分校） ； Vector Institute（向量研究所） ； Cardiff University（卡迪夫大学） ； University College London（伦敦大学学院） ； IIIT Hyderabad（海得拉巴印度理工学院） ； Harvard University（哈佛大学） ； Université de Montréal（蒙特利尔大学） ； Mila 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Knowledge-to-Jailbreak: Investigating Knowledge-driven Jailbreaking Attacks for Large Language Models

Shangqing Tu, Zhuoran Pan, Wenxuan Wang, Zhexin Zhang, Yuliang Sun, Jifan Yu, Hongning Wang, Lei Hou, Juanzi Li

机构 * Tsinghua Univerisity（清华大学） ； Peking University（北京大学） ； Hong Kong University of Science and Technology（香港理工大学） ； Beihang Univerisity（北京航空航天大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by KDD 2025 research track

Benchmarking large language models for materials synthesis: the case of atomic layer deposition

Angel Yanguas-Gil, Matthew T. Dearing, Jeffrey W. Elam, Jessica C. Jones, Sungjoon Kim, Adnan Mohammad, Chi Thang Nguyen, Bratin Sengupta

机构 * Applied Materials Division, Argonne National Laboratory（阿贡国家实验室应用材料部门） ； Business and Information Systems, Argonne National Laboratory（阿贡国家实验室商业与信息系统部门） ； Northwestern Center for Water Research, Northwestern University（西北大学水研究中心） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.AI、cs.LG

Audio-Aware Large Language Models as Judges for Speaking Styles

Cheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin, Linjie Li, Radu Kopetz, Yao Qian, Zhendong Wang, Zhengyuan Yang, Hung-yi Lee, Lijuan Wang

机构 * National Taiwan University（国立台湾大学） ； Microsoft（微软） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

Tingchen Fu, Mrinank Sharma, Philip Torr, Shay B. Cohen, David Krueger, Fazl Barez

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICML 2025. Tingchen Fu and Fazl Barez are core research contributors

CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation

Zhendong Mi, Renming Zheng, Haowen Zhong, Yue Sun, Seth Kneeland, Sayan Moitra, Ken Kutzer, Zhaozhuo Xu Shaoyi Huang

机构 * Stevens Institute of Technology（史蒂文斯理工学院） ； University of Washington（华盛顿大学） ； Amazon（亚马逊） ； SambaNova Systems（SambaNova系统） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；prompting(title,abstract)；分类 cs.AI、cs.LG

MMRefine: Unveiling the Obstacles to Robust Refinement in Multimodal Large Language Models

Gio Paik, Geewook Kim, Jinbae Im

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments ACL Findings 2025

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICML 2025

THELMA: Task Based Holistic Evaluation of Large Language Model Applications-RAG Question Answering

Udita Patel, Rutu Mulkar, Jay Roberts, Cibi Chakravarthy Senthilkumar, Sujay Gandhi, Xiaofei Zheng, Naumaan Nayyar, Parul Kalra, Rafael Castrillo

机构 * Amazon.com Services Inc.（亚马逊公司） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Added author

Who's Who: Large Language Models Meet Knowledge Conflicts in Practice

Quang Hieu Pham, Hoang Ngo, Anh Tuan Luu, Dat Quoc Nguyen

机构 * VinAI Research（VinAI研究院） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2024 Findings

Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?

Guillermo Marco, Julio Gonzalo, Ramón del Castillo, María Teresa Mateo Girona

机构 * School of Computer Science, UNED, Madrid, Spain（计算机科学学院，UNED，马德里，西班牙） ； Faculty of Education, UCM, Madrid, Spain（教育学院，UCM，马德里，西班牙） ； Faculty of Philosophy, UNED, Madrid, Spain（哲学学院，UNED，马德里，西班牙） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments 9 pages 6 figures

Journal ref Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19654-19670, Miami, Florida, USA. Association for Computational Linguistics

Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models

Haoyang Li, Xuejia Chen, Zhanchao XU, Darian Li, Nicole Hu, Fei Teng, Yiming Li, Luyu Qiu, Chen Jason Zhang, Qing Li, Lei Chen

机构 * Hong Kong Polytechnic University（香港理工大学） ； Hong Kong University of Science and Technology（香港科技大学） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2025

Representations of Fact, Fiction and Forecast in Large Language Models: Epistemics and Attitudes

Meng Li, Michael Vrazitulis, David Schlangen

机构 * University of Potsdam（波恩大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments accepted by ACL 2025 (main)

Adaptable Cardiovascular Disease Risk Prediction from Heterogeneous Data using Large Language Models

Frederike Lübeck, Jonas Wildberger, Frederik Träuble, Maximilian Mordig, Sergios Gatidis, Andreas Krause, Bernhard Schölkopf

机构 * Eidgenössische Technische Hochschule (ETH), Zürich, Switzerland（瑞士苏黎世联邦理工学院） ； Max Planck Institute for Intelligent Systems, Tübingen, Germany（德国图宾根智能系统研究所） ； ELLIS Institute, Tübingen, Germany（德国图宾根ELLIS研究所） ； Stanford School of Medicine, CA, USA（美国斯坦福医学院） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.AI、cs.LG

Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models

Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, Xuming Hu

机构 * Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）） ； Guangxi Zhuang Autonomous Region Big Data Research Institute（广西壮族自治区大数据研究院） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted by Findings of ACL 2025

Retrieval Models Aren't Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models

Zhengliang Shi, Yuhan Wang, Lingyong Yan, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, Zhaochun Ren

机构 * Shandong University（山东大学） ； Baidu Inc.（百度公司） ； Leiden University（莱顿大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments ACL 2025. Code: https://github.com/mangopy/tool-retrieval-benchmark

CCHall: A Novel Benchmark for Joint Cross-Lingual and Cross-Modal Hallucinations Detection in Large Language Models

Yongheng Zhang, Xu Liu, Ruoxi Zhou, Qiguang Chen, Hao Fei, Wenpeng Lu, Libo Qin

机构 * School of Computer Science and Engineering, Central South University, China（中南大学计算机科学与工程学院） ； National University of Singapore, Singapore（新加坡国立大学） 纠错

专题命中 评测与基准 ：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted at ACL 2025 Main Conference