On October 14th, 2020, researchers from OpenAI, the Stanford Institute for Human-Centered Artificial Intelligence, and other universities convened to discuss open research questions surrounding GPT-3, the largest publicly-disclosed dense language model at the time. The meeting took place under Chatham House Rules. Discussants came from a variety of research backgrounds including computer science, linguistics, philosophy, political science, communications, cyber policy, and more. Broadly, the discussion centered around two main questions: 1) What are the technical capabilities and limitations of large language models? 2) What are the societal effects of widespread use of large language models? Here, we provide a detailed summary of the discussion organized by the two themes above.
Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsense
University of Michigan（密歇根大学）；LG AI Research（LG AI研究院）；University of Illinois at Chicago（芝加哥大学伊利诺伊分校）；Singapore University of Technology Design（新加坡科技设计大学）
Carnegie Mellon University（卡内基梅隆大学）；University of Washington（华盛顿大学）；George Mason University（乔治梅森大学）
Recent advances in the capacity of large language models to generate human-like text have resulted in their increased adoption in user-facing settings. In parallel, these improvements have prompted a heated discourse around the risks of societal harms they introduce, whether inadvertent or malicious. Several studies have explored these harms and called for their mitigation via development of safer, fairer models. Going beyond enumerating the risks of harms, this work provides a survey of practical methods for addressing potential threats and societal harms from language generation models. We d…
MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation
University of Edinburgh（爱丁堡大学）；The University of Hong Kong（香港大学）；Harbin Institute of Technology（哈尔滨工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）；South China University of Technology（华南理工大学）；City University of Hong Kong（香港城市大学）；University of Cambridge（剑桥大学）
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
MIT（麻省理工学院）；Harvard University（哈佛大学）；Columbia University（哥伦比亚大学）；Cornell Tech（康奈尔科技校区）；Apollo Research（阿波罗研究院）；ETH Zurich（苏黎世联邦理工学院）；UC Berkeley（加州大学伯克利分校）；University of Sussex（萨塞克斯大学）；EffiSciences；University of Cambridge（剑桥大学）；New York University（纽约大学）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of Southern California（南加州大学）；Stanford University（斯坦福大学）
Can Large Language Models Master Complex Card Games?
Nankai University（南开大学）；Tsinghua University（清华大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Zhipu AI（智谱AI）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Multilingualism refers to the high degree of proficiency in two or more languages in the written and oral communication modes. It often results in language mixing, a.k.a. code-mixing, when a multilingual speaker switches between multiple languages in a single utterance of a text or speech. This paper discusses the current state of the NLP research, limitations, and foreseeable pitfalls in addressing five real-world applications for social good crisis management, healthcare, political campaigning, fake news, and hate speech for multilingual societies. We also propose futuristic datasets, models…
Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs
York University（约克大学）；Islamic University of Technology（伊斯兰科技大学）；University of Alberta（阿尔伯塔大学）；Dialpad Inc.（Dialpad公司）
Investigating the Capabilities and Limitations of Machine Learning for Identifying Bias in English Language Data with Information and Heritage Professionals
University of Edinburgh（爱丁堡大学）；Inria Bordeaux（波尔多信息与自动化研究院）
本文重新构建 ML 偏见问题，通过创建模型识别英语数据中的偏见而非消除它，并与信息与遗产专业人员合作评估，揭示了 ML 识别偏见的局限性，提出扩展 ML 公平性方法的混合研究路径。
CSAI, Tilburg University（蒂尔堡大学CSAI）；CLiPS, University of Antwerp（安特卫普大学CLiPS）；LT$ˆ3$, Ghent University（根特大学LT$ˆ3$）
The detection of online cyberbullying has seen an increase in societal importance, popularity in research, and available open data. Nevertheless, while computational power and affordability of resources continue to increase, the access restrictions on high-quality data limit the applicability of state-of-the-art techniques. Consequently, much of the recent research uses small, heterogeneous datasets, without a thorough evaluation of applicability. In this paper, we further illustrate these issues, as we (i) evaluate many publicly available resources for this task and demonstrate difficulties w…
Welfare Diplomacy: Benchmarking Language Model Cooperation
Stanford University（斯坦福大学）；Center on Long-Term Risk（长期风险研究中心）；UC Berkeley（加州大学伯克利分校）；University of Oxford（牛津大学）；Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）
The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systems
Computational Linguistics, Department of Linguistics, Bielefeld University（语言学计算系，语言学系，比勒菲尔德大学）；Honda Research Institute Europe（本田欧洲研究院）；Digital Linguistics Lab, Department of Linguistics, Bielefeld University（数字语言学实验室，语言学系，比勒菲尔德大学）
