We analyze the behaviors of open large language models (LLMs) on the task of data-to-text (D2T) generation, i.e., generating coherent and relevant text from structured data. To avoid the issue of LLM training data contamination with standard benchmarks, we design Quintd - a tool for collecting novel structured data records from public APIs. We find that open LLMs (Llama 2, Mistral, and Zephyr) can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd. However, we show that the semantic accuracy of the outputs is a major issue: both according to human annotators and our reference-free metric based on GPT-4, more than 80% of the outputs of open LLMs contain at least one semantic error. We publicly release the code, data, and model outputs.
Can Frontier LLMs Replace Annotators in Biomedical Text Mining? Analyzing Challenges and Exploring Solutions
The University of Tokyo(东京大学);Graduate School of Frontier Sciences(前沿科学研究生院);Department of Computational Biology and Medical Sciences(计算生物学与医学科学系);Database Center for Life Science(生命科学数据库中心);Joint Support-Center for Data Science Research(数据科学研究联合支持中心)
Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs
Research Centre for Information Systems Engineering (LIRIS), KU Leuven(信息系统工程研究中心(LIRIS),鲁汶大学);Department of Engineering Management, University of Antwerp(工程管理系,安特卫普大学);Department of Decision Analytics and Risk, University of Southampton(决策分析与风险系,南安普顿大学);School of Information, University of Michigan(信息学院,密歇根大学)
ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?
Department of Computer Science Northwestern University Evanston USA(计算机科学系西北大学艾文斯顿美国);Department of Computer Science University of Texas at Austin Austin USA(计算机科学系德克萨斯大学奥斯汀美国);Boston Children's Hospital, Harvard Medical School Boston USA(波士顿儿童医院哈佛医学院波士顿美国);Department of Computer Science Imperial College London London UK(计算机科学系伦敦帝国学院伦敦英国);Department of Computer Science Ohio State University Columbus USA(计算机科学系俄亥俄州立大学哥伦布美国);Massachusetts General Hospital, Harvard Medical School Boston USA(麻省总医院哈佛医学院波士顿美国);Department of Preventive Medicine, Feinberg School of Medicine Northwestern University Chicago USA(预防医学系费因伯格医学院西北大学芝加哥美国);Division of Computational Health Sciences, Department of Surgery University of Minnesota Minneapolis USA(计算健康科学部外科部明尼苏达大学明尼阿波利斯美国);Department of Population Health Sciences, Weill Cornell Medicine Cornell University New York USA(流行病学与公共卫生系韦尔·科恩医学中心康奈尔大学纽约美国);Department of Computer Science Emory University Atlanta USA(计算机科学系埃默里大学亚特兰大美国);Northwestern University(西北大学);University of Texas at Austin(德克萨斯大学奥斯汀);Boston Children's Hospital, Harvard Medical School(波士顿儿童医院哈佛医学院);Imperial College London(伦敦帝国学院);Ohio State University(俄亥俄州立大学);Massachusetts General Hospital, Harvard Medical School(麻省总医院哈佛医学院);University of Minnesota(明尼苏达大学);Cornell University(康奈尔大学);Emory University(埃默里大学)
To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity
LMU Munich(慕尼黑大学);Faculty of Philological and Cultural Studies, University of Vienna(维也纳大学语言与文化学院);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);UniVie Doctoral School Computer Science(维也纳大学计算机博士学院)
Enabling and Analyzing How to Efficiently Extract Information from Hybrid Long Documents with LLMs
School of Software & Microelectronics, Peking University(北京大学软件与微电子学院);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);Microsoft(微软公司);University of Technology Sydney(悉尼科技大学)
Analyzing Temporal Complex Events with Large Language Models? A Benchmark towards Temporal, Long Context Understanding
University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Singapore Management University(新加坡管理大学);Fudan University(复旦大学);National University of Singapore(新加坡国立大学)
Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
Rutgers University(罗格斯大学);Northwestern University(西北大学);UKP Lab, TU Darmstadt(德累斯顿技术大学UKP实验室);New Jersey Institute of Technology(新泽西理工学院);NEC Lab American(美国NEC实验室);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)
Analyzing Cancer Patients' Experiences with Embedding-based Topic Modeling and LLMs
Leiden Institute of Advanced Computer Science (LIACS), Leiden, The Netherlands(莱顿高级计算机科学研究所(LIACS),莱顿,荷兰);Leiden University Medical Center (LUMC), Leiden, The Netherlands(莱顿大学医学中心(LUMC),莱顿,荷兰)
When the LM misunderstood the human chuckled: Analyzing garden path effects in humans and language models
Blavatnik School of Computer Science, Tel Aviv University(特拉维夫大学布拉瓦尼克计算机科学学院);Department of Linguistics, Tel Aviv University(特拉维夫大学语言学系);Sagol School of Neuroscience, Tel Aviv University(特拉维夫大学萨戈尔神经科学学院)
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
EleutherAI;Booz Allen Hamilton(博思艾伦汉密尔顿);Yale University(耶鲁大学);University of Cambridge(剑桥大学);Indraprastha Institute of Information Technology Delhi(德里英迪拉普拉萨特信息技术学院);Stability AI;University of Amsterdam(阿姆斯特丹大学)