This chapter critically examines the potential contributions of modern language models to theoretical linguistics. Despite their focus on engineering goals, these models' ability to acquire sophisticated linguistic knowledge from mere exposure to data warrants a careful reassessment of their relevance to linguistic theory. I review a growing body of empirical evidence suggesting that language models can learn hierarchical syntactic structure and exhibit sensitivity to various linguistic phenomena, even when trained on developmentally plausible amounts of data. While the competence/performance distinction has been invoked to dismiss the relevance of such models to linguistic theory, I argue that this assessment may be premature. By carefully controlling learning conditions and making use of causal intervention methods, experiments with language models can potentially constrain hypotheses about language acquisition and competence. I conclude that closer collaboration between theoretical linguists and computational researchers could yield valuable insights, particularly in advancing debates about linguistic nativism.
Siebel School of Computing and Data Science(西贝尔计算与数据科学学院);The Grainger College of Engineering(格兰杰工程学院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Holmes: A Benchmark to Assess the Linguistic Competence of Language Models
Ubiquitous Knowledge Processing Lab (UKP Lab)(通用知识处理实验室);Technical University of Darmstadt(达姆施塔特技术大学);Information Systems Research Lab(信息系统研究实验室);Lucerne University of Applied Sciences and Arts(卢塞恩应用科学与艺术大学);IBM Research AI(IBM AI研究部);MIT CSAIL(MIT CSAIL实验室);MIT-IBM Watson AI Lab(MIT-IBM沃森AI实验室);IBM Research Europe - Ireland(IBM欧洲爱尔兰研究部)
Dissociating language and thought in large language models
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Georgia Institute of Technology(佐治亚理工学院);University of California Los Angeles(加利福尼亚大学洛杉矶分校);Massachusetts Institute of Technology(麻省理工学院)
Evaluating the Deductive Competence of Large Language Models
Wright State University(莱特州立大学);Oak Ridge Institute for Science and Education(橡树岭科学与教育研究所);Air Force Research Laboratory(空军研究实验室);AI Institute - University of South Carolina(南卡罗来纳大学人工智能研究所)
Benchmarking Motivational Interviewing Competence of Large Language Models
National Institute of Mental Health and Neuro Sciences (NIMHANS)(国家精神卫生与神经科学研究所(NIMHANS));Indian Institute of Technology (IIT) Kharagpur(印度理工学院卡拉格普尔分校(IIT Kharagpur))
BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian
Stanford University(斯坦福大学);Freie Universität Berlin(柏林自由大学);Center for Information and Language Processing, LMU Munich(慕尼黑大学信息与语言处理中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Leibniz Supercomputing Centre (LRZ)(莱布尼茨超级计算中心)
Unveiling Linguistic Regions in Large Language Models
School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心)
Evaluation of Cultural Competence of Vision-Language Models
Department of Computer Science, University of Copenhagen(计算机科学系,哥本哈根大学);Department of Rhetoric and Communication Studies, University of Richmond(修辞与传播研究系,里士满大学);Department of Data Science and Statistics, University of Richmond(数据科学与统计系,里士满大学);Google Research(谷歌研究);ILLC, University of Amsterdam(阿姆斯特丹大学ILLC中心);Pioneer Centre of AI, Denmark(丹麦先锋人工智能中心)
From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens
University of Waterloo(滑铁卢大学);Vector Institute(向量研究所);University of Michigan(密歇根大学);Stony Brook University(石溪大学);McGill University(麦吉尔大学);Dartmouth College(达特茅斯学院)
Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?
NASK National Research Institute(NASK国家研究所);Adam Mickiewicz University(亚当·密茨凯维奇大学);ARAAI Poland(ARAAI波兰);Poznań University of Medical Sciences(波兹南医科大学);Centre of Postgraduate Medical Education, Poland(波兰研究生医学教育中心);T. Marciniak Lower Silesian Specialist Hospital(T. 马尔奇尼亚克下西里西亚专科医院);Medical University of Warsaw(华沙医科大学)
Polishing Every Facet of the GEM: Testing Linguistic Competence of LLMs and Humans in Korean
Department of Artificial Intelligence, Korea University, Seoul, South Korea(韩国大学人工智能系);Department of Computer Science and Engineering, Korea University, Seoul, South Korea(韩国大学计算机科学与工程系);Institute for Digital HUSS, Korea University, Seoul, South Korea(韩国大学数字HUSS研究所)
Left-right asymmetry in predicting brain activity from LLMs' representations emerges with their formal linguistic competence
Centre d’Analyse et de Mathématique Sociales CNRS, EHESS, Paris, France(分析与数学社会中心 CNRS,EHESS,巴黎,法国);Cognitive Neuroimaging Unit CNRS, INSERM, CEA, Neurospin Center, 91191 Gif-sur-Yvette, France(认知神经成像单元 CNRS,INSERM,CEA,Neurospin中心,法国91191 Gif-sur-Yvette)
Research Borderlands: Analysing Writing Across Research Cultures
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校Siebel计算与数据科学学院);Pioneer Centre for AI, University of Copenhagen(哥本哈根大学先锋人工智能中心);Allen Institute for Artificial Intelligence (Ai2)(人工智能研究院)