A Survey on Large Language Models with some Insights on their Capabilities and Limitations
Expedia Group（亿客行集团）；Roma Tre University（罗马第三大学）
Andrea Matarazzo, Riccardo Torlone
备注：
174 pages, to be submitted to a journal in a shorter version. It includes figures taken from papers by other authors. All the sources have been referenced. arXiv admin note: text overlap with arXiv:2303.18223 by other authors
The rapid advancement of artificial intelligence, particularly with the development of Large Language Models (LLMs) built on the transformer architecture, has redefined the capabilities of natural language processing. These models now exhibit remarkable performance across various language-related tasks, such as text generation, question answering, translation, and summarization, often rivaling human-like comprehension. More intriguingly, LLMs have demonstrated emergent abilities extending beyond their core functions, showing proficiency in tasks like commonsense reasoning, code generation, and arithmetic. This survey paper explores the foundational components, scaling mechanisms, and architectural strategies that drive these capabilities. Emphasizing models like GPT and LLaMA, we analyze the impact of exponential data and computational growth on LLM performance, while also addressing the trade-offs associated with scaling. We also examine LLM applications across sectors, such as healthcare, finance, education, and law, highlighting their adaptability and potential to solve domain-specific challenges. Central to this work are the questions of how LLMs generalize across diverse tasks, exhibit planning, and reasoning abilities, and whether these emergent abilities can be systematically elicited or enhanced. In particular, we provide some insights into the CoT (Chain of Thought) and PoT (Plan of Thought) abilities within LLMs, focusing on how pre-training data influences their emergence. Additionally, we investigate LLM-modulo frameworks that integrate external systems, allowing LLMs to handle complex, dynamic tasks. By analyzing these factors, this paper aims to foster the ongoing discussion on the capabilities and limits of LLMs, promoting their responsible development and application in novel and increasingly complex environments.
On October 14th, 2020, researchers from OpenAI, the Stanford Institute for Human-Centered Artificial Intelligence, and other universities convened to discuss open research questions surrounding GPT-3, the largest publicly-disclosed dense language model at the time. The meeting took place under Chatham House Rules. Discussants came from a variety of research backgrounds including computer science, linguistics, philosophy, political science, communications, cyber policy, and more. Broadly, the discussion centered around two main questions: 1) What are the technical capabilities and limitations o…
We analyze the memory capacity of a delay based reservoir computer with a Hopf normal form as nonlinearity and numerically compute the linear as well as the higher order recall capabilities. A possible physical realisation could be a laser with external cavity, for which the information is fed via electrical injection. A task independent quantification of the computational capability of the reservoir system is done via a complete orthonormal set of basis functions. Our results suggest that even for constant readout dimension the total memory capacity is dependent on the ratio between the infor…
Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense
University of Michigan（密歇根大学）；LG AI Research（LG AI研究院）；University of Illinois at Chicago（芝加哥大学伊利诺伊分校）；Singapore University of Technology Design（新加坡科技设计大学）
MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation
University of Edinburgh（爱丁堡大学）；The University of Hong Kong（香港大学）；Harbin Institute of Technology（哈尔滨工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）；South China University of Technology（华南理工大学）；City University of Hong Kong（香港城市大学）；University of Cambridge（剑桥大学）
Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models
Chandar Research Lab（Chandar研究实验室）；Mila – Quebec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Microsoft Research（微软研究院）；Polytechnique Montréal（蒙特利尔理工学院）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
On the Limitations of Steering in Language Model Alignment
Birla Institute of Technology and Science Pilani（比拉理工科学与 Pilani 学院）；National University of Singapore（新加坡国立大学）；Institute for Data Science, National University of Singapore（新加坡国立大学数据科学研究院）
Generative Pre-trained Transformers (GPTs) have recently been scaled to unprecedented sizes in the history of machine learning. These models, solely trained on the language modeling objective, have been shown to exhibit outstanding few-shot learning capabilities in a number of different tasks. Nevertheless, aside from anecdotal experiences, little is known regarding their multilingual capabilities, given the fact that the pre-training corpus is almost entirely composed of English text. In this work, we investigate the multilingual skills of GPT-3, focusing on one language that barely appears i…
Is LLM an Overconfident Judge? Unveiling the Capabilities of LLMs in Detecting Offensive Language with Annotation Disagreement
Key Laboratory of Social Computing and Cognitive Intelligence, Dalian University of Technology（大连理工大学社会计算与认知智能重点实验室）；School of Computer Science and Technology, Xinjiang Normal University（新疆师范大学计算机科学与技术学院）；Social AI Studio, Singapore University of Technology and Design（新加坡科技设计大学社会AI工作室）
Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)
School of Computer Science and Engineering, Southeast University, Nanjing, China（计算机科学与工程学院，东南大学，南京，中国）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其跨学科应用关键实验室（东南大学），教育部，中国）
Can large language models assist choice modelling? Insights into prompting strategies and current models' capabilities
Choice Modelling Centre & Institute for Transport Studies, University of Leeds（选择建模中心及交通研究学院，利兹大学）；CityAI Lab, Transport and Logistics group, Engineering Systems and Services Department, TU Delft（CityAI实验室，运输与物流组，工程系统与服务部门，代尔夫特理工大学）
Exploring the Limitations of Kolmogorov-Arnold Networks in Classification: Insights to Software Training and Hardware Implementation
Nara Institute of Science and Technology（奈良科学技术研究所）；University of Information Technology - VNUHCM（胡志明市国家大学信息学院）；Le Quy Don technical University（黎贵敦工业大学）
An Insight into Security Code Review with LLMs: Capabilities, Obstacles, and Influential Factors
School of Computer Science, Wuhan University, China（武汉大学计算机科学学院）；School of Mathematical and Computational Sciences, Massey University（梅西大学数学与计算科学学院）；School of Computing Technologies, RMIT University, Australia（皇家墨尔本理工学院计算技术学院）
