语言大模型 / LLM

AIonopedia: an LLM agent orchestrating multimodal learning for ionic liquid discovery

Yuqi Yin, Yibo Fu, Siyuan Wang, Peng Sun, Hongyu Wang, Xiaohui Wang, Lei Zheng, Zhiyong Li, Zhirong Liu, Jianji Wang, Zhaoxi Sun

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；foundation model(abstract)

OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking

Yanhong Li, Tianyang Xu, Kenan Tang, Karen Livescu, David McAllester, Jiawei Zhou

机构 * University of Chicago（芝加哥大学） ； TTI-Chicago（芝加哥技术研究所） ； UC Santa Barbara（圣巴巴拉大学） ； Stony Brook University（石溪大学） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；pretraining(abstract)

Curiosity-Driven LLM-as-a-judge for Personalized Creative Judgment

Vanya Bannihatti Kumar, Divyanshu Goyal, Akhil Eppa, Neel Bhandari

机构 * Adobe Inc.（Adobe公司） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；SFT(abstract)

Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports

Punit Kumar Singh, Nishant Kumar, Akash Ghosh, Kunal Pasad, Khushi Soni, Manisha Jaishwal, Sriparna Saha, Syukron Abu Ishaq Alfarozi, Asres Temam Abagissa, Kitsuchart Pasupa, Haiqin Yang, Jose G Moreno

机构 * Indian Institute of Technology Patna（印度理工学院帕纳布分校） ； Sardar Patel Institute of Technology（萨达尔·帕特尔技术学院） ； Universitas Gadjah Mada（加查马大学） ； King Mongkut’s Institute of Technology Ladkrabang（拉差班国王技术学院） ； Shenzhen Technology University（深圳技术大学） ； Université de Toulouse（图卢兹大学） 纠错

专题命中 评测与基准 ：language model(title,abstract)；large language model(abstract)；small language model(abstract)；prompting(abstract)

Comments 52 pages, 56 figures; appearing at EMNLP'25

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks

Hongchao Jiang, Yiming Chen, Yushi Cao, Hung-yi Lee, Robby T. Tan

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Dataset is available at https://huggingface.co/datasets/mattymchen/codejudgebench

Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs

Sai Krishna Mendu, Harish Yenala, Aditi Gulati, Shanu Kumar, Parag Agrawal

机构 * Microsoft（微软） 纠错

专题命中 评测与基准 ：pretraining(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments 10 pages, 5 figures. Accepted at the International Joint Conferences on Artificial Intelligence IJCAI 2025 (main track)

Evaluating Podcast Recommendations with Profile-Aware LLM-as-a-Judge

Francesco Fabbri, Gustavo Penha, Edoardo D'Amico, Alice Wang, Marco De Nadai, Jackie Doremus, Paul Gigioli, Andreas Damianou, Oskar Stal, Mounia Lalmas

机构 * Spotify 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Accepted at RecSys '25

The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs

Nitay Calderon, Roi Reichart, Rotem Dror

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Optimizing LLM-Based Multi-Agent System with Textual Feedback: A Case Study on Software Development

Ming Shen, Raphael Shu, Anurag Pratik, James Gung, Yubin Ge, Monica Sunkara, Yi Zhang

机构 * Arizona State University（亚利桑那州立大学） ； Amazon Web Services（亚马逊网络服务） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

From Words to Collisions: LLM-Guided Evaluation and Adversarial Generation of Safety-Critical Driving Scenarios

Yuan Gao, Mattia Piccinini, Korbinian Moller, Amr Alanwar, Johannes Betz

机构 * Professorship of Autonomous Vehicle Systems, TUM School of Engineering and Design, Technical University of Munich（自主车辆系统教授职位，技术大学慕尼黑工程与设计学院） ； Munich Institute of Robotics and Machine Intelligence (MIRMI)（慕尼黑机器人与机器智能研究所） ； TUM School of Computation, Information and Technology, Department of Computer Engineering, Technical University of Munich（技术大学慕尼黑计算、信息与技术学院，计算机工程系） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Final Version and Paper Accepted at IEEE ITSC 2025

CaseReportBench: An LLM Benchmark Dataset for Dense Information Extraction in Clinical Case Reports

Xiao Yu Cindy Zhang, Carlos R. Ferreira, Francis Rossignol, Raymond T. Ng, Wyeth Wasserman, Jian Zhu

机构 * University of British Columbia（不列颠哥伦比亚大学） ； National Institutes of Health（美国国家卫生研究院） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Next Token Perception Score: Analytical Assessment of your LLM Perception Skills

Yu-Ang Cheng, Leyang Hu, Hai Huang, Randall Balestriero

机构 * Atlassian（Atlassian公司） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；pretraining(abstract)

Unmasking the Canvas: A Dynamic Benchmark for Image Generation Jailbreaking and LLM Content Safety

Variath Madhupal Gautham Nair, Vishal Varma Dantuluri

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

To FP8 and Back Again: Quantifying Reduced Precision Effects on LLM Training Stability

Joonhyung Lee, Jeongin Bae, Byeongwook Kim, Se Jung Kwon, Dongsoo Lee

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；pretraining(abstract)

Process-based Self-Rewarding Language Models

Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, Yeyun Gong

专题命中 评测与基准 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；preference optimization(abstract)

Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking

Samar M. Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Abdelfadil, Shady Shehata, Muhammad Abdul-Mageed

专题命中 评测与基准 ：LLM(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Project GitHub page is accessible at: https://github.com/UBC-NLP/jawaher

Prompting Techniques for Secure Code Generation: A Systematic Investigation

Catherine Tony, Nicolás E. Díaz Ferreyra, Markus Mutas, Salem Dhiff, Riccardo Scandariato

专题命中 评测与基准 ：prompting(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Work partially supported by the EU-funded project Sec4AI4Sec: Cybersecurity for AI-Augmented Systems (grant no. 101120393) - ACCEPTED at ACM Transactions on Software Engineering and Methodology (Feb. 2025)

Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods

Jai Doshi, Asa Cooper Stickland

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments 9 pages, 2 figures

LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient

Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, Kan Li

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Towards Understanding the Robustness of LLM-based Evaluations under Perturbations

Manav Chaudhary, Harshit Gupta, Savita Bhat, Vasudeva Varma

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Accepted at ICON 2024

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free

Ziyue Li, Tianyi Zhou

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Code: https://github.com/tianyi-lab/MoE-Embedding

From Persona to Personalization: A Survey on Role-Playing Language Agents

Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, Aili Chen, Nianqi Li, Lida Chen, Caiyu Hu, Siye Wu, Scott Ren, Ziquan Fu, Yanghua Xiao

专题命中 评测与基准 ：language agent(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Accepted to TMLR 2024

R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, Gongshen Liu

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments EMNLP Findings 2024

RGD: Multi-LLM Based Agent Debugger via Refinement and Generation Guidance

Haolin Jin, Zechao Sun, Huaming Chen

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

EmPO: Emotion Grounding for Empathetic Response Generation through Preference Optimization

Ondrej Sotolar, Vojtech Formanek, Alok Debnath, Allison Lahnala, Charles Welch, Lucie FLek

专题命中 评测与基准 ：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments v02, 8 pages long paper, EMNLP ACL style

Zyda: A 1.3T Dataset for Open Language Modeling

Yury Tokpanov, Beren Millidge, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, Quentin Anthony

专题命中 评测与基准 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；pretraining(abstract)

Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting

Halley Young, Yimeng Zeng, Jacob Gardner, Osbert Bastani

专题命中 评测与基准 ：prompting(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case

Sonia Meyer, Shreya Singh, Bertha Tam, Christopher Ton, Angel Ren

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

BADGE: BADminton report Generation and Evaluation with LLM

Shang-Hsuan Chiang, Lin-Wei Chao, Kuang-Da Wang, Chih-Chuan Wang, Wen-Chih Peng

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Accepted by IJCAI 2024 Workshop: The 2nd International Workshop on Intelligent Technologies for Precision Sports Science (IT4PSS)

Input Conditioned Graph Generation for Language Agents

Lukas Vierling, Jie Fu, Kai Chen

专题命中 评测与基准 ：language agent(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)