Large language models (LLMs) are currently at the forefront of intertwining artificial intelligence (AI) systems with human communication and everyday life. Thus, aligning them with human values is of great importance. However, given the steady increase in reasoning abilities, future LLMs are under suspicion of becoming able to deceive human operators and utilizing this ability to bypass monitoring efforts. As a prerequisite to this, LLMs need to possess a conceptual understanding of deception strategies. This study reveals that such strategies emerged in state-of-the-art LLMs, such as GPT-4, but were non-existent in earlier LLMs. We conduct a series of experiments showing that state-of-the-art LLMs are able to understand and induce false beliefs in other agents, that their performance in complex deception scenarios can be amplified utilizing chain-of-thought reasoning, and that eliciting Machiavellianism in LLMs can alter their propensity to deceive. In sum, revealing hitherto unknown machine behavior in LLMs, our study contributes to the nascent field of machine psychology.
This research revisits the classic Turing test and compares recent large language models such as ChatGPT for their abilities to reproduce human-level comprehension and compelling text generation. Two task challenges -- summarization, and question answering -- prompt ChatGPT to produce original content (98-99%) from a single text entry and also sequential questions originally posed by Turing in 1950. We score the original and generated content against the OpenAI GPT-2 Output Detector from 2019, and establish multiple cases where the generated content proves original and undetectable (98%). The …
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；University of Chicago（芝加哥大学）；The Chinese University of Hong Kong（香港中文大学）；East China Normal University（华东师范大学）
The Impact of Off-Policy Training Data on Probe Generalisation
King’s College London（伦敦国王学院）；Imperial College London（伦敦帝国学院）；LASR Labs（LASR实验室）；University of Manchester（曼彻斯特大学）；Goodfire AI；University of Cambridge（剑桥大学）
Large neural language models trained on massive amounts of text have emerged as a formidable strategy for Natural Language Understanding tasks. However, the strength of these models as Natural Language Generators is less clear. Though anecdotal evidence suggests that these models generate better quality text, there has been no detailed study characterizing their generation abilities. In this work, we compare the performance of an extensively pretrained model, OpenAI GPT2-117 (Radford et al., 2019), to a state-of-the-art neural story generation model (Fan et al., 2018). By evaluating the genera…
Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist
Xi’an Jiaotong-Liverpool University（西交利物浦大学）；University of Liverpool（利物浦大学）；University of Macau（澳门大学）；HKUST（香港科技大学）；Microsoft Research Asia（微软亚洲研究院）；Duke Kunshan University（昆山杜克大学）
Large language models (LLMs) have emerged as valuable tools for many natural language understanding tasks. In safety-critical applications such as healthcare, the utility of these models is governed by their ability to generate outputs that are factually accurate and complete. In this work, we present dialog-enabled resolving agents (DERA). DERA is a paradigm made possible by the increased conversational abilities of LLMs, namely GPT-4. It provides a simple, interpretable forum for models to communicate feedback and iteratively improve output. We frame our dialog as a discussion between two ag…
HEART-Bench: Do LLM Agents Exhibit Human-like Psychology?
Shanghai Jiao Tong University（上海交通大学）；Imperial College London（伦敦帝国理工学院）；Quwan Group（启元集团）；University of Washington（华盛顿大学）；South China Normal University（华南师范大学）
GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院）；Institute of Natural Sciences, Shanghai Jiao Tong University（上海交通大学自然科学研究院）
BLINK: Multimodal Large Language Models Can See but Not Perceive
University of Pennsylvania（宾夕法尼亚大学）；University of Washington（华盛顿大学）；Allen Institute for AI（艾伦人工智能研究所）；University of California, Davis（加利福尼亚大学戴维斯分校）；Columbia University（哥伦比亚大学）
From Instruction to Output: The Role of Prompting in Modern NLG
Faculty of Information Technology, Monash University（莫纳什大学信息科技学院）；School of Computer, Data and Mathematical Sciences, Western Sydney University（西澳大学计算机、数据与数学科学学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）
Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?
AIML Lab, TU Darmstadt（AIML实验室，德累斯顿技术大学）；Hessian Center for AI (hessian.ai)（黑森人工智能中心）；Institute of Psychology, TU Darmstadt（心理学研究所，德累斯顿技术大学）；Centre for Cognitive Science, TU Darmstadt（认知科学中心，德累斯顿技术大学）；German Center for AI (DFKI)（德国人工智能中心）
Leveraging Large Language Model as Simulated Patients for Clinical Education
Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Computer Science, Wuhan University（武汉大学计算机学院）；CureFun Co.（CureFun公司）
KARPA: A Training-free Method of Adapting Knowledge Graph as References for Large Language Model's Reasoning Path Aggregation
Beijing University of Posts and Telecommunications（北京邮电大学）；Tongji University（同济大学）；Multimodal Art Projection Research Community（多模态艺术投影研究社区）；Duke University（杜克大学）；ZTE Corporation（中兴通讯股份有限公司）
