语言大模型 / LLM - arXivDaily 专题

2506.05692 2025-06-23 cs.CR cs.AI 85%

SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, Xinchen Gu

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

2506.15253 2025-06-19 cs.CR cs.AI 85%

RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments

Yuchuan Fu, Xiaohan Yuan, Dongxia Wang

机构 * Zhejiang University（浙江大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments 12 pages, 8 figures

2506.14809 2025-06-19 cs.HC cs.LG 85%

Impact of a Deployed LLM Survey Creation Tool through the IS Success Model

Peng Jiang, Vinicius Cezar Monteiro de Lira, Antonio Maiorino

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

2506.14808 2025-06-19 cs.LG 85%

PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models

Jenny Schmalfuss, Nadine Chang, Vibashan VS, Maying Shen, Andres Bruhn, Jose M. Alvarez

机构 * University of Stuttgart（斯图加特大学）； NVIDIA（NVIDIA公司）； Johns Hopkins University（约翰霍普金斯大学）

专题命中评测与基准：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.LG

Comments Accepted to CVPR 2025

2506.13639 2025-06-17 cs.CL 85%

An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability

Yusuke Yamauchi, Taro Yano, Masafumi Oyamada

机构 * The University of Tokyo（东京大学）； NEC Corporation（日本电讯公司）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2502.15266 2025-06-17 cs.CL 85%

A Training-free LLM-based Approach to General Chinese Character Error Correction

Houquan Zhou, Bo Zhang, Zhenghua Li, Ming Yan, Min Zhang

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Accepted at Main Conference of ACL 2025, 26 pages, 12 figures

2506.11007 2025-06-16 cs.SE cs.AI 85%

Impact of Comments on LLM Comprehension of Legacy Code

Rock Sabetto, Emily Escamilla, Devesh Agarwal, Sujay Kandwal, Justin F. Brunelle, Scott Rosen, Nitin Naik, Samruddhi Thaker, Eric O. Scott, Jacob Zimmer, Amit Madan, Arun Sridharan, Doug Wendt, Michael Doyle, Christopher Glasz, Jasper Phillips, William Macke, Colin Diggs, Michael Bartholf, Zachary Robin, Paul Ursino

机构 * The MITRE Corporation（MITRE公司）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

2409.15724 2025-06-13 cs.SE cs.AI cs.IR 85%

LLM-Cure: LLM-based Competitor User Review Analysis for Feature Enhancement

Maram Assi, Safwat Hassan, Ying Zou

机构 * Université du Québec à Montréal（魁北克大学蒙特利尔分校）； University of Toronto（多伦多大学）； Queen’s University（皇后大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments 25 pages

Journal ref ACM Trans. Softw. Eng. Methodol. (June 2025)

2506.10829 2025-06-13 cs.CY cs.AI 85%

LLM-Driven Personalized Answer Generation and Evaluation

Mohammadreza Molavi, Mohammadreza Tavakoli, Mohammad Moein, Abdolali Faraji, Gábor Kismihók

机构 * Leibniz Information Centre for Science and Technology（莱布尼茨科学与技术信息中心）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments This is the preprint version of a paper accepted at AIED 2025. The final version will be published by Springer

2503.13572 2025-06-13 cs.AR cs.CR cs.LG 85%

VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination

Zeng Wang, Minghao Shao, Jitendra Bhandari, Likhitha Mankali, Ramesh Karri, Ozgur Sinanoglu, Muhammad Shafique, Johann Knechtel

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

2410.11802 2025-06-13 cs.LG 85%

TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting

Zhe Li, Xiangfei Qiu, Peng Chen, Yihang Wang, Hanyin Cheng, Yang Shu, Jilin Hu, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Bin Yang

机构 * East China Normal University（东华师范大学）； Aalborg University（奥尔堡大学）

专题命中评测与基准：foundation model(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

2506.09359 2025-06-12 cs.CL 85%

Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQL

Qingyun Zeng, Simin Ma, Arash Niknafs, Ashish Basran, Carol Szabo

机构 * Microsoft Copilot Studio AI（微软Copilot工作室）； Zoom Communications（Zoom通讯）； University of Pennsylvania（宾夕法尼亚大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments 8 pages

2506.05981 2025-06-11 cs.AI 85%

CrimeMind: Simulating Urban Crime with Multi-Modal LLM Agents

Qingbin Zeng, Ruotong Zhao, Jinzhu Mao, Haoyang Li, Fengli Xu, Yong Li

机构 * Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）； Department of Sociology, Hong Kong Baptist University（香港 Baptist 大学社会学系）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments Typos corrected

2506.07947 2025-06-10 cs.CL 85%

Statistical Hypothesis Testing for Auditing Robustness in Language Models

Paulius Rauba, Qiyao Wei, Mihaela van der Schaar

机构 * University of Cambridge（剑桥大学）

专题命中评测与基准：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL

Comments arXiv admin note: substantial text overlap with arXiv:2412.00868

Journal ref Forty-second International Conference on Machine Learning. ICML 2025

2506.07731 2025-06-10 cs.AI 85%

NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models

Mouadh Yagoubi, Yasser Dahou, Billel Mokeddem, Younes Belkada, Phuc H. Le-Khac, Basma El Amel Boussaha, Reda Alami, Jingwei Zuo, Damiano Marsili, Mugariya Farooq, Mounia Lalmas, Georgia Gkioxari, Patrick Gallinari, Philip Torr, Hakim Hacid

机构 * Technology Innovation Institute（技术创新研究所）； California Institute of Technology（加州理工学院）； Spotify ； ISIR - Sorbonne University（ISIR - 巴黎 Sorbonne 大学）； University of Oxford（牛津大学）

专题命中评测与基准：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.AI

2506.07596 2025-06-10 cs.LG 85%

TwinBreak: Jailbreaking LLM Security Alignments based on Twin Prompts

Torsten Krauß, Hamid Dashtbani, Alexandra Dmitrienko

机构 * University of Würzburg（乌尔姆大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

Comments 26 pages, 25 tables, 13 figures, 2 algorithms, to appear in the 43th USENIX Security Symposium (USENIX Security 2025)

2506.07402 2025-06-10 cs.CR cs.CL 85%

Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures

Yukai Zhou, Sibei Yang, Wenjie Wang

机构 * ShanghaiTech University（上海科技大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2503.14996 2025-06-10 cs.CL 85%

Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering

Francesco Maria Molfese, Luca Moroni, Luca Gioffré, Alessandro Scirè, Simone Conia, Roberto Navigli

机构 * Sapienza University of Rome（罗马萨皮恩扎大学）； Babelscape

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Findings of the Association for Computational Linguistics ACL 2025

2503.04381 2025-06-09 cs.CL 85%

TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge

Cheng-Han Chiang, Hung-yi Lee, Michal Lukasik

机构 * NTU GICE（国立台湾大学人工智能研究中心）； Google Research（谷歌研究）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments ACL 2025 camera-ready Codes and models are available at https://github.com/d223302/TRACT

2410.22318 2025-06-09 cs.LG 85%

Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting

Can Chen, Jun-Kun Wang

机构 * University of California San Diego（加州大学圣地亚哥分校）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

2506.05073 2025-06-06 cs.CL 85%

Just a Scratch: Enhancing LLM Capabilities for Self-harm Detection through Intent Differentiation and Emoji Interpretation

Soumitra Ghosh, Gopendra Vikram Singh, Shambhavi, Sabarna Choudhury, Asif Ekbal

机构 * Fondazione Bruno Kessler（布鲁诺·克瑟勒基金会）； Indian Institute of Technology Patna（印度理工学院帕纳布分校）； Indian Institute of Technology Jodhpur（印度理工学院朱罗普尔分校）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments To be published in the Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025 Main)

2506.04894 2025-06-06 cs.CL 85%

ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests

Shiyi Xu, Yiwen Hu, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Ji-Rong Wen

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2504.17934 2025-06-06 cs.HC cs.CL cs.CR 85%

Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents

Chaoran Chen, Zhiping Zhang, Ibrahim Khalilov, Bingcan Guo, Simret A Gebreegziabher, Yanfang Ye, Ziang Xiao, Yaxing Yao, Tianshi Li, Toby Jia-Jun Li

机构 * University of Notre Dame（圣约翰大学）； Northeastern University（东北大学）； Virginia Tech（弗吉尼亚理工大学）； University of Washington（华盛顿大学）； Johns Hopkins University（约翰霍普金斯大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2402.12649 2025-06-06 cs.CL stat.AP 85%

Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation

Kristian Lum, Jacy Reese Anthis, Kevin Robinson, Chirag Nagpal, Alexander D'Amour

机构 * University of Chicago（芝加哥大学）； Stanford University（斯坦福大学）； Google Research（谷歌研究）； Google DeepMind（谷歌DeepMind）

专题命中评测与基准：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL

Comments Published in ACL 2025

2506.02442 2025-06-05 cs.CL 85%

Should LLM Safety Be More Than Refusing Harmful Instructions?

Utsav Maskey, Mark Dras, Usman Naseem

机构 * Macquarie University（麦考瑞大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Preprint

2504.05276 2025-06-05 cs.CL 85%

Enhancing LLM-Based Short Answer Grading with Retrieval-Augmented Generation

Yucheng Chu, Peng He, Hang Li, Haoyu Han, Kaiqi Yang, Yu Xue, Tingting Li, Joseph Krajcik, Jiliang Tang

机构 * Michigan State University（密歇根州立大学）； Washington State University（华盛顿州立大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments EDM 2025 Short Paper

2506.02696 2025-06-04 cs.AI 85%

Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations

Jinyuan Luo, Zhen Fang, Yixuan Li, Seongheon Park, Ling Chen

机构 * Australian Artificial Intelligence Institute, University of Technology Sydney（澳大利亚人工智能研究所，悉尼技术大学）； Department of Computer Sciences, University of Wisconsin-Madison（计算机科学系，威斯康星大学麦迪逊分校）

专题命中评测与基准：LLM(title)；large language model(abstract)；language model(abstract)；prompting(abstract)

2411.13899 2025-06-03 cs.LG cs.AR 85%

Schemato -- An LLM for Netlist-to-Schematic Conversion

Ryoga Matsuo, Stefan Uhlich, Arun Venkitaraman, Andrea Bonetti, Chia-Yu Hsieh, Ali Momeni, Lukas Mauch, Augusto Capone, Eisaku Ohbuchi, Lorenzo Servadei

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

2505.16978 2025-06-03 cs.AI cs.PL 85%

HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation

Weizhi Tang, Yixuan Li, Chris Sypherd, Elizabeth Polgreen, Vaishak Belle

机构 * University of Edinburgh（爱丁堡大学）

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments Accepted to ACL 2025 Findings. Code available at https://github.com/RutaTang/HyGenar

2412.13602 2025-06-03 cs.CL 85%

GAMEBoT: Transparent Assessment of LLM Reasoning in Games

Wenye Lin, Jonathan Roberts, Yunhan Yang, Samuel Albanie, Zongqing Lu, Kai Han

专题命中评测与基准：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments 9 pages, ACL 2025

1. 评测与基准 32599 篇

SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments

Impact of a Deployed LLM Survey Creation Tool through the IS Success Model

PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models

An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability

A Training-free LLM-based Approach to General Chinese Character Error Correction

Impact of Comments on LLM Comprehension of Legacy Code

LLM-Cure: LLM-based Competitor User Review Analysis for Feature Enhancement

LLM-Driven Personalized Answer Generation and Evaluation

VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination

TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting

Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQL

CrimeMind: Simulating Urban Crime with Multi-Modal LLM Agents

Statistical Hypothesis Testing for Auditing Robustness in Language Models

NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models

TwinBreak: Jailbreaking LLM Security Alignments based on Twin Prompts

Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures

Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering

TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge

Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Betting

Just a Scratch: Enhancing LLM Capabilities for Self-harm Detection through Intent Differentiation and Emoji Interpretation

ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests

Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents

Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation

Should LLM Safety Be More Than Refusing Harmful Instructions?

Enhancing LLM-Based Short Answer Grading with Retrieval-Augmented Generation

Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations

Schemato -- An LLM for Netlist-to-Schematic Conversion

HyGenar: An LLM-Driven Hybrid Genetic Algorithm for Few-Shot Grammar Generation

GAMEBoT: Transparent Assessment of LLM Reasoning in Games