Large language models (LLMs) are the result of a massive experiment in bottom-up, data-driven reverse engineering of language at scale. Despite their utility in a number of downstream NLP tasks, ample research has shown that LLMs are incapable of performing reasoning in tasks that require quantification over and the manipulation of symbolic variables (e.g., planning and problem solving); see for example [25][26]. In this document, however, we will focus on testing LLMs for their language understanding capabilities, their supposed forte. As we will show here, the language understanding capabilities of LLMs have been widely exaggerated. While LLMs have proven to generate human-like coherent language (since that's how they were designed), their language understanding capabilities have not been properly tested. In particular, we believe that the language understanding capabilities of LLMs should be tested by performing an operation that is the opposite of 'text generation' and specifically by giving the LLM snippets of text as input and then querying what the LLM "understood". As we show here, when doing so it will become apparent that LLMs do not truly understand language, beyond very superficial inferences that are essentially the byproduct of the memorization of massive amounts of ingested text.
Alignment Revisited: Are Large Language Models Consistent in Stated and Revealed Preferences?
Department of Information Systems and Business Analytics(信息系统与商业分析系);University at Albany - State University of New York(阿尔巴尼大学 - 新 york 状态大学);Santa Clara, CA(圣克拉拉,加州);Princeton Junction, NJ(普林斯顿 junction,新泽西)
An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analyses
Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW Sydney(过渡人工智能研究组,数学与统计学学院,UNSW悉尼);Department of Civil Engineering, National Institute of Technology, Rourkela, India(土木工程系,印度国家理工学院,拉尔凯拉);Centre for Artificial Intelligence and Innovation, Pingla Institute, Sydney, Australia(人工智能与创新中心,Pingla研究所,悉尼,澳大利亚)
The Knowledge-Reasoning Dissociation: Fundamental Limitations of LLMs in Clinical Natural Language Inference
Department of Computer Science, University of Manchester, UK(曼彻斯特大学计算机科学系);National Biomarker Centre, CRUK-MI, University of Manchester, UK(曼彻斯特大学国家生物标志物中心);Idiap Research Institute, Switzerland(日内瓦IDIAP研究所);School of Computer Science, University of Sheffield, UK(谢菲尔德大学计算机科学学院)
Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization
Eindhoven University of Technology(埃因霍温理工大学);Brandenburg University of Technology(勃兰登堡工业大学);Nanyang Technological University(南洋理工大学);Delft University of Technology(代尔夫特理工大学);Southeast University(东南大学)
CLNX: Bridging Code and Natural Language for C/C++ Vulnerability-Contributing Commits Identification
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Hubei Key Laboratory of Distributed System Security(湖北省分布式系统安全重点实验室);Hubei Engineering Research Center on Big Data Security(湖北省大数据安全工程研究中心)
Confusion is the Final Barrier: Rethinking Jailbreak Evaluation and Investigating the Real Misuse Threat of LLMs
State Key Lab of Processors, Institute of Computing Technology, CAS(中国科学院计算技术研究所状态关键实验室);University of Chinese Academy of Sciences(中国科学院大学);People’s Public Security University of China(中国人民公安大学);Chinese University of Hong Kong(香港大学)
NativQA: Multilingual Culturally-Aligned Natural Query for LLMs
University of New Brunswick(新不伦瑞克大学);Qatar Computing Research Institute(卡塔尔计算研究中心);UPES(UPES大学);Carnegie Mellon University in Qatar(卡塔尔卡内基梅隆大学);Qatar University(卡塔尔大学)
InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information
School of Cyber Science and Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Key Laboratory of Cyberspace Security, Ministry of Education(教育部网络空间安全重点实验室);Hubei Key Laboratory of Distributed System Security(湖北省分布式系统安全重点实验室)
How Does Naming Affect LLMs on Code Analysis Tasks?
College of Information Science and Technology, The Pennsylvania State University(宾夕法尼亚州立大学信息科学与技术学院);Heinz College, Carnegie Mellon University(卡内基梅隆大学海因茨学院)
Consistent but Miscalibrated: Evaluating LLM Limitations for Risk Communication in Natural Language
Faculty of Dental Medicine and Oral Health Sciences, McGill University(麦吉尔大学牙医学与口腔健康科学学院);Chandar Research Lab, Polytechnique Montréal(蒙特利尔理工大学钱达尔研究实验室);Mila – Québec Artificial Intelligence Institute(魁北克人工智能研究所米拉);Département de Génie Informatique et Génie Logiciel (GIGL), Polytechnique Montréal(蒙特利尔理工大学计算机科学与软件工程系)
QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language
Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Pengcheng Laboratory(鹏城实验室);The Chinese University of Hong Kong(香港中文大学);Jilin University(吉林大学);Southwest University(西南大学);University of Electronic Science and Technology of China(电子科技大学);Shenzhen University(深圳大学)
Do Models Hear Like Us? Probing the Representational Alignment of Audio LLMs and Naturalistic EEG
School of Computer Science, Chongqing University(重庆大学计算机科学学院);Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua University(清华大学计算机科学与技术系、人工智能研究院);School of Economics and Business Administration, Chongqing University(重庆大学经济与商业管理学院);School of Artificial Intelligence, Southwest University(西南大学人工智能学院);The First Affiliated Hospital of Chongqing Medical University(重庆医科大学第一附属医院)
Exploring Conversational Design Choices in LLMs for Pedagogical Purposes: Socratic and Narrative Approaches for Improving Instructor's Teaching Practice