The Uniform Information Density (UID) principle posits that humans prefer to spread information evenly during language production. We examine if this UID principle can help capture differences between Large Language Models (LLMs)-generated and human-generated texts. We propose GPT-who, the first psycholinguistically-inspired domain-agnostic statistical detector. This detector employs UID-based features to model the unique statistical signature of each LLM and human author for accurate detection. We evaluate our method using 4 large-scale benchmark datasets and find that GPT-who outperforms state-of-the-art detectors (both statistical- & non-statistical) such as GLTR, GPTZero, DetectGPT, OpenAI detector, and ZeroGPT by over $20$% across domains. In addition to better performance, it is computationally inexpensive and utilizes an interpretable representation of text articles. We find that GPT-who can distinguish texts generated by very sophisticated LLMs, even when the overlying text is indiscernible. UID-based measures for all datasets and code are available at https://github.com/saranya-venkatraman/gpt-who.
Whose LLM is it Anyway? Linguistic Comparison and LLM Attribution for GPT-3.5, GPT-4 and Bard
Department of Information Science(信息科学系);Bar Ilan University(巴伊兰大学);Department of Information Systems(信息系统系);University of Haifa(海法大学);Department of Computing(计算机系);Jonkoping University(约翰·科普林大学)
Tell Me Who Your Students Are: GPT Can Generate Valid Multiple-Choice Questions When Students' (Mis)Understanding Is Hinted
Tohoku University(东北大学);The University of Electro-Communications(电气通信大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);North Carolina State University(北卡罗来纳州立大学);RIKEN(日本理化学研究所)
Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);DIRO, Université de Montréal(蒙特利尔大学 DIRO 学院);Kuaishou(快手);School of Information, Renmin University of China(中国人民大学信息学院)
Large Language Models can be Guided to Evade AI-Generated Text Detection
Southern University of Science and Technology(南方科技大学);Hong Kong University of Science and Technology(香港科技大学);Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局);Nanyang Technological University (NTU)(南洋理工大学)
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
The Chinese University of Hong Kong(香港中文大学);Tianjin Medical University(天津医科大学);Tencent AI Lab(腾讯AI实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Biasless Language Models Learn Unnaturally: How LLMs Fail to Distinguish the Possible from the Impossible
Tel Aviv University(特拉维夫大学);Ecole Normale Supérieure(巴黎高等师范学院);Earth Species Project(地球物种项目);EHESS, PSL University, CNRS(法国社会科学高等研究院、巴黎文理研究大学、法国国家科学研究中心)
Says Who? Effective Zero-Shot Annotation of Focalization
Department of Computer Science, Cornell University(计算机科学系,康奈尔大学);Center for Humanities Computing, Aarhus University(人文学计算中心,奥胡斯大学);Department of Linguistics, Cognitive Science, and Semiotics, Aarhus University(语言学、认知科学与符号学系,奥胡斯大学);TEXT - Center for the Contemporary Cultures of Text, Aarhus University(文本当代文化中心,奥胡斯大学)
Use Me Wisely: AI-Driven Assessment for LLM Prompting Skills Development
Università degli Studi di Milano Bicocca(米兰比可卡大学);University of Regensburg(雷根斯堡大学);Universitat Pompeu Fabra(庞培法布拉大学);Politecnico di Milano(米兰理工大学);National Research Council of Italy (CNR)(意大利国家研究委员会)
DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents
Curai Health(Curai健康)
大语言模型(LLM)已经成为许多自然语言理解任务的有价值的工具。在医疗保健等安全关键型应用中,这些模型的实用性取决于它们生成真实准确和完整的输出的能力。在这项工作中,我们提出了对话使能解析代理(DERA)。DERA是一个范例,通过LLM的会话能力的增加而成为可能,即GPT-4。它为模型提供了一个简单的、可解释的论坛,以交流反馈并迭代地改进输出。我们框架我们的对话作为两个代理类型之间的讨论-一个研究员,谁处理信息,并确定关键的问题组件,和一个决策者,谁拥有自主权,以整合研究员的信息,并作出判断的最终输出。我们测试DERA对三个临床重点任务。对于医疗会话摘要和护理计划生成,DERA在人类专家偏好评估和定量指标方面都显示出比基本GPT-4性能的显着改进。在一项新的发现中,我们还表明,GPT-4在MedQA问答(QA)数据集(Jin et al. 2021,USMLE)的开放式版本上的性能(70%)远高于及格水平(60%),DERA表现出类似的性能。我们在https://github.com/curai/curai-research/tree/main/DERA上发布开放式MEDQA数据集。
The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
Vanderbilt University(范德堡大学);UK AI Safety Institute(英国人工智能安全研究所);Apollo Research(阿波罗研究院);New York University(纽约大学);University of Sussex(萨塞克斯大学);University of Oxford(牛津大学)
Can AI Debias the News? LLM Interventions Improve Cross-Partisan Receptivity but LLMs Overestimate Their Own Effectiveness
Department of Experimental Psychology, University of Oxford(心理学实验系,牛津大学);Uehiro Oxford Institute, University of Oxford(牛津大学欧胡罗研究所);Psychology Programme, School of Social Sciences, Nanyang Technological University(南洋理工大学社会科学学院心理学项目);Department of Communication and Culture, BI Norwegian Business School(BI挪威商学院传播与文化系)
Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations
The University of Texas Health Science Center at Houston(德克萨斯大学休斯顿健康科学中心);Yale University(耶鲁大学);Mayo Clinic(妙佑医疗国际);China Academy of Chinese Medical Sciences(中国中医科学院);The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心)
本文以 WHO 标准的 361 个穴位、3,174 个定位关系为语料,比较 GPT 与 LSTM、BioBERT 的关系抽取性能,发现微调 GPT-3.5 表现最佳,微平均 F1 达 0.92。