语言大模型 / LLM - arXivDaily 专题

2312.06674 2023-12-13 cs.CL cs.AI 86%

Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, Madian Khabsa

专题命中评测与基准：LLM(title,abstract)；language model(abstract)；prompting(abstract)；分类 cs.CL、cs.AI

2311.08592 2023-12-01 cs.SE cs.AI cs.CL 86%

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

Bhaktipriya Radharapu, Kevin Robinson, Lora Aroyo, Preethi Lahoti

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2310.16111 2023-12-01 cs.CL cs.CR cs.LG 86%

Locally Differentially Private Document Generation Using Zero Shot Prompting

Saiteja Utpala, Sara Hooker, Pin Yu Chen

专题命中评测与基准：prompting(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at EMNLP 2023 (Findings)

2311.17295 2023-11-30 cs.CL cs.AI 86%

Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

Meriem Boubdir, Edward Kim, Beyza Ermis, Sara Hooker, Marzieh Fadaee

专题命中评测与基准：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments 22 pages, 7 figures, 2 tables. Revised version of the paper accepted at GEM Workshop, EMNLP 2023

2311.04850 2023-11-14 cs.CL cs.AI 86%

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples

Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica

专题命中评测与基准：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

2311.02049 2023-11-06 cs.CL cs.AI 86%

Post Turing: Mapping the landscape of LLM Evaluation

Alexey Tikhonov, Ivan P. Yamshchikov

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted for GEM @ EMNLP 2023

2311.01964 2023-11-06 cs.CL cs.AI 86%

Don't Make Your LLM an Evaluation Benchmark Cheater

Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, Jiawei Han

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 11 pages

2310.13469 2023-10-25 cs.CL cs.AI 86%

Ask Language Model to Clean Your Noisy Translation Data

Quinten Bolding, Baohao Liao, Brandon James Denis, Jun Luo, Christof Monz

专题命中评测与基准：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2023, Findings

2310.09265 2023-10-16 cs.CL cs.LG 86%

PromptRE: Weakly-Supervised Document-Level Relation Extraction via Prompting-Based Data Programming

Chufan Gao, Xulin Fan, Jimeng Sun, Xuan Wang

专题命中评测与基准：prompting(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

2309.13701 2023-09-28 cs.CL cs.AI cs.HC 86%

ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning

Hosein Hasanbeig, Hiteshi Sharma, Leo Betthauser, Felipe Vieira Frujeri, Ida Momennejad

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2309.10677 2023-09-28 cs.CL cs.AI 86%

Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation

Yucheng Li

专题命中评测与基准：language model(title,abstract)；large language model(abstract)；foundation model(abstract)；分类 cs.CL、cs.AI

2309.04369 2023-09-11 cs.CL cs.AI 86%

Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation

Jiatong Li, Rui Li, Qi Liu

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

2307.15008 2023-07-28 cs.CR cs.AI cs.LG 86%

A LLM Assisted Exploitation of AI-Guardian

Nicholas Carlini

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

2307.11864 2023-07-25 cs.SI cs.CL cs.CR cs.LG 86%

The Looming Threat of Fake and LLM-generated LinkedIn Profiles: Challenges and Opportunities for Detection and Prevention

Navid Ayoobi, Sadat Shahriar, Arjun Mukherjee

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 33rd ACM Conference on Hypertext and Social Media (HT '23)

2307.08074 2023-07-25 cs.CL cs.AI 86%

Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling

Longyue Wang, Zefeng Du, Donghuai Liu, Deng Cai, Dian Yu, Haiyun Jiang, Yan Wang, Leyang Cui, Shuming Shi, Zhaopeng Tu

专题命中评测与基准：language model(title,abstract)；large language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments Zhaopeng Tu is the corresponding author

2306.03314 2023-06-07 cs.AI cs.LG cs.MA 86%

Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

Yashar Talebirad, Amirhossein Nadiri

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

2306.01116 2023-06-05 cs.CL cs.AI 86%

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay

专题命中评测与基准：LLM(title)；large language model(abstract)；language model(abstract)；pretraining(abstract)

2607.18086 2026-07-21 cs.AI 新提交 86%

Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

临床大语言模型中证据充分性提示的依赖判断的安全增益和特定模型的有用性成本

Koyar Afrasyab

专题命中评测与基准：prompting(title)；LLM(abstract,abstract_cn)；language model(abstract)；分类 cs.AI

AI总结研究临床大语言模型中证据充分性提示的安全增益及有用性成本，通过在公共数据基准中让四个模型用标准提示和包装器回答问题，发现安全增益有方向和幅度差异且依赖评判者，同时存在模型特定的有用性成本。

Comments https://github.com/KAVentures/clinical-evidence-sufficiency-llm

2606.00815 2026-06-02 cs.LG 86%

OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models

OmniEEG-Bench: 脑电图基础模型的标准化评估基准

Ziling Lu, Zongsheng Li, Xinke Shen, Kexin Lou, Yingyue Xin, Xiaoqi Chen, Shinan Wang, Xiang Chen, Jiahao Fan, Chenyu Huang, Xin Xu, Zhoujie Hou, Chen Wei, Quanying Liu

机构 * Department of Biomedical Engineering, Southern University of Science and Technology, Shenzhen, China（南方科技大学生物医学工程系，深圳，中国）； School of Computer Science and Engineering, The Chinese University of Hong Kong, Shenzhen, China（香港中文大学（深圳）计算机科学与工程学院，深圳，中国）； Omni-Intelligence, Shenzhen, China（奥米智能，深圳，中国）； Shenzhen Loop Area Institute, Shenzhen, China（深圳环城研究院，深圳，中国）

专题命中评测与基准：foundation model(title,abstract)；LLM(abstract)；pretraining(abstract)；分类 cs.LG

AI总结针对脑电图基础模型评估碎片化问题，提出统一基准OmniEEG-Bench，涵盖六类任务、54个数据集，并揭示预训练数据多样性和模型大小与性能的缩放律关系。

Comments 28 pages, 13 figures, 8 tables; benchmark of EEG foundation models

2509.26351 2026-03-31 cs.LG 86%

LLM-Assisted Emergency Triage Benchmark: Bridging Hospital-Rich and MCI-Like Field Simulation

基于大语言模型的急救分级基准：连接医院丰富与MCI类现场模拟

Joshua Sebastian, Karma Tobden, KMA Solaiman

机构 * University of Maryland Baltimore County（马里兰大学巴尔的摩县分校）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

AI总结本文提出一个开放的急救分级基准，通过大语言模型辅助构建，解决现有基准不可用的问题，涵盖医院环境和MCI模拟两种场景，提升临床AI数据集的可重复性和可访问性。

Comments Submitted to GenAI4Health@NeurIPS 2025. This was the first version of the LLM-assisted emergency triage benchmark dataset and baseline models. A related but separate benchmark-focused study on emergency triage under constrained sensing has been accepted at the IEEE International Conference on Healthcare Informatics (ICHI) 2026 (see arXiv:2602.20168)

2601.11903 2026-01-21 cs.AI 86%

AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systems

AEMA：可验证评估框架用于可信和受控的代理LLM系统

YenTing Lee, Keerthi Koneru, Zahra Moslemi, Sheethal Kumar, Ramesh Radhakrishnan

机构 * University of California, San Diego（加州大学圣地亚哥分校）； Center for Advanced AI, Accenture（Accenture高级人工智能中心）； University of California, Irvine（加州大学伊拉斯姆斯分校）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

AI总结 AEMA提出了一种可验证的评估框架，用于评估基于LLM的多代理系统，通过人类监督实现稳定、可追溯的自动化评估。

Comments Workshop on W51: How Can We Trust and Control Agentic AI? Toward Alignment, Robustness, and Verifiability in Autonomous LLM Agents at AAAI 2026

2512.00461 2025-12-02 cs.CY cs.CL 86%

Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

谁的人设？LLM研究中的人设实验及透明化路径

Jan Batzner, Volker Stocker, Bingjun Tang, Anusha Natarajan, Qinhao Chen, Stefan Schmid, Gjergji Kasneci

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

AI总结本文探讨了LLM研究中合成人设实验的代表性与生态效度问题，提出透明化检查表以提升评估的严谨性和实证性。

Comments Published at AAAI/ACM AIES 2025. Presented at NeurIPS 2025 Workshop Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling

Journal ref Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 8(1), 2025, 343-354

2510.05909 2025-10-21 cs.AI 86%

Optimizing for Persuasion Improves LLM Generalization: Evidence from Quality-Diversity Evolution of Debate Strategies

Aksel Joonas Reedi, Corentin Léger, Julien Pourcel, Loris Gaven, Perrine Charriau, Guillaume Pourcel

机构 * University of Groningen（格罗宁根大学）； Inria, Flowers team（法国国家信息与自动化研究所，Flowers团队）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments Open-source code available at https://github.com/flowersteam/llm_persuasion

2508.16889 2025-10-10 cs.CL 86%

ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks

Hyunjun Kim, Junwoo Ha, Sangyoon Yu, Haon Park

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments NeurIPS 2025 Workshop on MTI-LLM

2510.01295 2025-10-03 cs.AI cs.MA 86%

The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation

Zarreen Reza

机构 * Independent researcher（独立研究者）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments 39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling

2509.18658 2025-09-24 cs.CL 86%

Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction

Huanxin Sheng, Xinyi Liu, Hangfeng He, Jieyu Zhao, Jian Kang

机构 * University of Rochester（罗切斯特大学）； University of Southern California（南加州大学）； MBZUAI

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments To appear in EMNLP 2025. Our code and data are available at \url{https://github.com/BruceSheng1202/Analyzing_Uncertainty_of_LLM-as-a-Judge

2509.13978 2025-09-24 cs.DC cs.AI cs.DB 86%

LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology

Renan Souza, Timothy Poteet, Brian Etz, Daniel Rosendo, Amal Gueroudji, Woong Shin, Prasanna Balaprakash, Rafael Ferreira da Silva

机构 * Oak Ridge National Lab.（橡树岭国家实验室）； Argonne National Lab.（阿贡国家实验室）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments Paper accepted in the proceedings of the Supercomputing Conference (SC). Cite it as Renan Souza, Timothy Poteet, Brian Etz, Daniel Rosendo, Amal Gueroudji, Woong Shin, Prasanna Balaprakash, and Rafael Ferreira da Silva. LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology. In WORKS at the ACM/IEEE International Conference on Supercomputing, 2025

2509.15356 2025-09-22 cs.LG 86%

Predicting Language Models' Success at Zero-Shot Probabilistic Prediction

Kevin Ren, Santiago Cortes-Gomez, Carlos Miguel Patiño, Ananya Joshi, Ruiqi Lyu, Jingjing Tang, Alistair Turcan, Khurram Yamin, Steven Wu, Bryan Wilder

机构 * Cornell Tech（康奈尔科技）； Carnegie Mellon University（卡内基梅隆大学）

专题命中评测与基准：language model(title,abstract)；LLM(abstract,comments)；large language model(abstract)；分类 cs.LG

Comments EMNLP Findings 2025. We release our code at: camera-ready" target="_blank" rel="noopener">https://github.com/kkr36/llm-eval/tree/camera-ready

2507.16372 2025-07-23 cs.CR cs.AI 86%

Depth Gives a False Sense of Privacy: LLM Internal States Inversion

Tian Dong, Yan Meng, Shaofeng Li, Guoxing Chen, Zhen Liu, Haojin Zhu

机构 * Shanghai Jiao Tong University（上海交通大学）； Southeast University（东南大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments Accepted by USENIX Security 2025. Please cite this paper as "Tian Dong, Yan Meng, Shaofeng Li, Guoxing Chen, Zhen Liu, Haojin Zhu. Depth Gives a False Sense of Privacy: LLM Internal States Inversion. In the 34th USENIX Security Symposium (USENIX Security '25)."

2507.15815 2025-07-22 cs.MA cs.LG 86%

LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra

Seth Karten, Wenzhe Li, Zihan Ding, Samuel Kleiner, Yu Bai, Chi Jin

机构 * Princeton University（普林斯顿大学）； Salesforce Research（Salesforce研究）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

Comments 27 pages, 6 figures, Code: https://github.com/sethkarten/LLM-Economist

1. 评测与基准 32413 篇

Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

Locally Differentially Private Document Generation Using Zero Shot Prompting

Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples

Post Turing: Mapping the landscape of LLM Evaluation

Don't Make Your LLM an Evaluation Benchmark Cheater

Ask Language Model to Clean Your Noisy Translation Data

PromptRE: Weakly-Supervised Document-Level Relation Extraction via Prompting-Based Data Programming

ALLURE: Auditing and Improving LLM-based Evaluation of Text using Iterative In-Context-Learning

Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation

Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluation

A LLM Assisted Exploitation of AI-Guardian

The Looming Threat of Fake and LLM-generated LinkedIn Profiles: Challenges and Opportunities for Detection and Prevention

Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling

Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs

OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models

LLM-Assisted Emergency Triage Benchmark: Bridging Hospital-Rich and MCI-Like Field Simulation

AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systems

Whose Personae? Synthetic Persona Experiments in LLM Research and Pathways to Transparency

Optimizing for Persuasion Improves LLM Generalization: Evidence from Quality-Diversity Evolution of Debate Strategies

ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks

The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation

Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction

LLM Agents for Interactive Workflow Provenance: Reference Architecture and Evaluation Methodology

Predicting Language Models' Success at Zero-Shot Probabilistic Prediction

Depth Gives a False Sense of Privacy: LLM Internal States Inversion

LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra