语言大模型 / LLM

Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards

Furkan Şahinuç, Thy Thy Tran, Yulia Grishina, Yufang Hou, Bei Chen, Iryna Gurevych

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Improving Large Language Models in Event Relation Logical Prediction

Meiqi Chen, Yubo Ma, Kaitao Song, Yixin Cao, Yan Zhang, Dongsheng Li

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.AI

Comments ACL 2024

Analysis of Argument Structure Constructions in the Large Language Model BERT

Pegah Ramezani, Achim Schilling, Patrick Krauss

专题命中 评测与基准 ：language model(title,abstract)；large language model(title)；分类 cs.CL

Comments arXiv admin note: text overlap with arXiv:2408.03062

Command-line Obfuscation Detection using Small Language Models

Vojtech Outrata, Michael Adam Polak, Martin Kopp

专题命中 评测与基准 ：language model(title,abstract)；small language model(title)；分类 cs.LG

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Comments 36 pages, 6 figures, 13 tables (ACL 2024 Findings)

Fine-grained Gender Control in Machine Translation with Large Language Models

Minwoo Lee, Hyukhun Koh, Minsung Kim, Kyomin Jung

专题命中 评测与基准 ：large language model(title)；language model(title)；prompting(abstract)；分类 cs.CL

Comments NAACL 2024 Main track long paper

Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, Lori Levin

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Comments LREC-COLING 2024

Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations

Guan-Ting Lin, Cheng-Han Chiang, Hung-yi Lee

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Comments Accepted by ACL 2024

MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks

Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng, Rishav Hada, Prachi Jain, Maxamed Axmed, Kalika Bali, Sunayana Sitaram

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Comments 40 pages, 35 figures and 34 tables

Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Healthcare Professionals

Avishek Choudhury, Zaria Chaudhry

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.AI

Comments 1 figure

WaterJudge: Quality-Detection Trade-off when Watermarking Large Language Models

Piotr Molenda, Adian Liusie, Mark J. F. Gales

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Comments NAACL 2024 (Findings)

Dia-LLaMA: Towards Large Language Model-driven CT Report Generation

Zhixuan Chen, Luyang Luo, Yequan Bie, Hao Chen

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.AI

Comments 10 pages

SUQL: Conversational Search over Structured and Unstructured Data with Large Language Models

Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina J. Semnani, Chen Jie Yu, Monica S. Lam

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Scope of Large Language Models for Mining Emerging Opinions in Online Health Discourse

Joseph Gatto, Madhusudan Basak, Yash Srivastava, Philip Bohlman, Sarah M. Preum

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

LLM on FHIR -- Demystifying Health Records

Paul Schmiedmayer, Adrit Rao, Philipp Zagar, Vishnu Ravi, Aydin Zahedivash, Arash Fereydooni, Oliver Aalami

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract,comments)；language model(abstract,comments)；分类 cs.AI

Comments Pre-print of the paper submitted to the Call for Papers for the Special Focus Issue on ChatGPT and Large Language Models (LLMs) in Biomedicine and Health at the Journal of the American Medical Informatics Association: https://academic.oup.com/jamia/pages/call-for-papers-for-special-focus-issue

ReSLLM: Large Language Models are Strong Resource Selectors for Federated Search

Shuai Wang, Shengyao Zhuang, Bevan Koopman, Guido Zuccon

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.AI

Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models

Tian Liang, Zhiwei He, Jen-tse Huang, Wenxuan Wang, Wenxiang Jiao, Rui Wang, Yujiu Yang, Zhaopeng Tu, Shuming Shi, Xing Wang

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

Comments Work in progress

User Simulation with Large Language Models for Evaluating Task-Oriented Dialogue

Sam Davidson, Salvatore Romeo, Raphael Shu, James Gung, Arshit Gupta, Saab Mansour, Yi Zhang

专题命中 评测与基准 ：language model(title,abstract)；large language model(title)；分类 cs.CL

Comments 13 pages

Large Language Models are Diverse Role-Players for Summarization Evaluation

Ning Wu, Ming Gong, Linjun Shou, Shining Liang, Daxin Jiang

专题命中 评测与基准 ：large language model(title)；language model(title)；prompting(abstract)；分类 cs.CL

Comments NLPCC 2023

Prompt me a Dataset: An investigation of text-image prompting for historical image dataset creation using foundation models

Hassan El-Hajj, Matteo Valleriani

专题命中 评测与基准 ：foundation model(title,abstract)；prompting(title)；分类 cs.AI

Comments 12 pages, 3 figures, Accepted in ICIAP2023, AI4DH workshop

ArcGPT: A Large Language Model Tailored for Real-world Archival Applications

Shitou Zhang, Jingrui Hou, Siyuan Peng, Zuchao Li, Qibiao Hu, Ping Wang

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract)；分类 cs.CL

DISCO: Distilling Counterfactuals with Large Language Models

Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, Kyle Richardson

专题命中 评测与基准 ：language model(title,abstract)；large language model(title)；分类 cs.CL

Comments ACL 2023 camera ready, final title change

Accelerating Large Language Model Decoding with Speculative Sampling

Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, John Jumper

专题命中 评测与基准 ：language model(title,abstract)；large language model(title)；分类 cs.CL

A Systematic Investigation of Commonsense Knowledge in Large Language Models

Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d'Autume, Phil Blunsom, Aida Nematzadeh

专题命中 评测与基准 ：language model(title,abstract)；large language model(title)；分类 cs.CL

Comments Accepted to EMNLP 2022

SLING: Sino Linguistic Evaluation of Large Language Models

Yixiao Song, Kalpesh Krishna, Rajesh Bhatt, Mohit Iyyer

专题命中 评测与基准 ：language model(title,abstract)；large language model(title)；分类 cs.CL

Comments 29 pages, EMNLP 2022 camera ready

Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba

专题命中 评测与基准 ：language model(title,abstract)；large language model(title)；分类 cs.LG

Comments corrected typos, added references, added authors, added acknowledgements

Large Language Models Enable Automated Formative Feedback in Human-Robot Interaction Tasks

Emily Jensen, Sriram Sankaranarayanan, Bradley Hayes

专题命中 评测与基准 ：large language model(title)；language model(title)；LLM(abstract,comments)

Comments Presented at Human-LLM Interaction Workshop at HRI 2024

Distinct Profiles of Run-to-Run Score Reliability and Expert-Panel Alignment Across Four LLM Evaluators of Simulated Japanese-Language AI-to-AI Counseling

固定合成日语咨询对话中的结构化提示与自动评估

Keita Kiuchi, Yoshikazu Fujimoto, Hideyuki Gotō, Tomonori Hosokawa, Makoto Nishimura, Yōsuke Satō, Izumi Sezai, Tomohiro Inoue

机构 * Japan National Institute of Occupational Safety and Health（日本国立职业安全卫生研究所） ； Kaze To Taiyo（凯泽・太阳） ； Saga Occupational Health Association（Saga职业健康协会） ； Department of Pharmacy, Zikei Hospital/Zikei Institute of Psychiatry（药剂科，Zikei医院/Zikei精神医学研究所） ； Department of Medical Welfare, Suzuka University of Medical Science（医疗福祉科， Suzuka医科大学） ； Graduate School of Human Sciences, Ritsumeikan University（人类科学研究生院，立命馆大学） ； Faculty of Nursing, National Defense Medical College（护理学部，国家防卫医疗大学） ； Support Center for Students with Disabilities, Aoyama Gakuin University（残疾学生支持中心，上智大学） 纠错

专题命中 评测与基准 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

AI总结 研究通过人工智能生成的18份固定日语咨询对话记录，对比GPT - minimal、GPT - SMDP和Claude - SMDP三种情况，咨询专家与新的语言模型分别评级，发现SMDP对话在多方面获更高专家评级，语言模型评级可重复但偏宽松。

Comments 55 pages, 2 figures, 31 tables; supplemental material included; data and code at https://doi.org/10.5281/zenodo.22106028; preregistration and amendments at https://doi.org/10.17605/OSF.IO/VU286

Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

风格取胜，实质落败：对作为创意生成评判者的大语言模型（LLM）的诊断

Fengxian Ji, Yuke Li, Jingpu Yang, Juanfan Wu, Fan Zhang, Zhexuan Cui, Yu Xie, Min Peng, Qianqian Xie, Xiuying Chen, Zhuohan Xie

专题命中 评测与基准 ：LLM(title,title_cn)；分类 cs.CL、cs.AI

AI总结 本研究针对大语言模型作为创意评判者的风格偏差问题，提出SciStyleBench基准及配套的SciStyleExtractor模块，实验验证该模块可有效降低风格偏差、提升实质区分度，为科学创意评估提供了系统性解决方案。

Comments First three authors are co-first authors

Evaluation Awareness in Language Models: Representation, Verbalization, and Control

语言模型中的评估感知：表示、语言化与控制

Farzaneh Heidari, Amin Memarian, Guillaume Rabusseau

专题命中 评测与基准 ：language model(title,abstract)；LLM(abstract,abstract_cn)；分类 cs.CL、cs.AI

AI总结 该研究系统探测了6个语言模型的评估感知，发现其可线性解码、与语言化部分对齐，且在Olmo模型中随微调放大，需评估时考虑模型内部表示与语言化的脱节。

Comments The first two authors contributed equally