大型语言模型(LLMs)会从多个文档来源中记忆文本。在预训练阶段,LLM 训练目标是最大化文本的似然概率,但既不接收文本的来源信息,也不记忆来源。因此,LLM 无法提供生成内容的文档信息,用户也无法获得任何关于可靠性(这对事实性或隐私侵犯至关重要)的提示。这项工作在解码步骤中引入了词元级别的来源识别,将词元表示映射到参考文档。我们提出了一种双词元来源识别器,它是一个多层感知器,输入为两个连续的词元表示,以实现更好的泛化。我们在 Wikipedia 和 PG19 数据集上进行了广泛的实验,涉及多种 LLM、层位置和识别器规模。总体结果表明,词元级别的来源识别器在追踪文档方面具有可能性,这是安全使用 LLM 的关键问题。
英文摘要:
Large language models (LLMs) memorize text from several sources of documents. In pretraining, LLM trains to maximize the likelihood of text but neither receives the source of the text nor memorizes the source. Accordingly, LLM can not provide document information on the generated content, and users do not obtain any hint of reliability, which is crucial for factuality or privacy infringement. This work introduces token-level source identification in the decoding step, which maps the token representation to the reference document. We propose a bi-gram source identifier, a multi-layer perceptron with two successive token representations as input for better generalization. We conduct extensive experiments on Wikipedia and PG19 datasets with several LLMs, layer locations, and identifier sizes. The overall results show a possibility of token-level source identifiers for tracing the document, a crucial problem for the safe use of LLMs.
Identifying the Periodicity of Information in Natural Language
CLCS Lab, Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系,南方科技大学计算机语言实验室);Digital Linguistics Lab, Department of Linguistics, Bielefeld University(语言学系,比勒菲尔德大学数字语言学实验室)
Exploiting ChatGPT for Diagnosing Autism-Associated Language Disorders and Identifying Distinct Features
University at Albany(奥尔巴尼大学);West Virginia University(西弗吉尼亚大学);Washington University in St. Louis(圣路易斯华盛顿大学);Camas High School(卡马斯高中);California Institute of Technology(加州理工学院)
Identifying and Decomposing Compound Ingredients in Meal Plans Using Large Language Models
Faculty of Health Sciences, University of Maribor(马里博尔大学健康科学学院);Faculty of Electrical Engineering and Computer Science, University of Maribor(马里博尔大学电气工程与计算机科学学院);The Nu B.V.(The Nu有限公司);Pain Research & Intervention Center of Excellence (PRICE), University of Florida(佛罗里达大学疼痛研究与干预卓越中心(PRICE));Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida(佛罗里达大学医学院健康结果与生物医学信息学系);Usher Institute, University of Edinburgh(爱丁堡大学厄舍研究所)
A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals
Linkalab s.r.l.(Linkalab公司);Department of Mathematics and Computer Science(数学与计算机科学系);University of Cagliari(卡利亚里大学);Knowledge Media Institute(知识媒体研究所);The Open University(开放大学)
Who Wrote This? Identifying Machine vs Human-Generated Text in Hausa
Kalinga University(卡林加大学);Bayero University(巴耶罗大学);Arewa Data Science Academy(阿雷瓦数据科学学院);DSFSI, University of Pretoria(比勒陀利亚大学 DSFSI);Northeastern University(东北大学);Imperial College London(伦敦帝国学院)
Identifying and Mitigating Social Bias Knowledge in Language Models
Zhejiang University(浙江大学);Nanyang Technological University(南洋理工大学);Angelalign Technology Inc.(时代天使科技有限公司);A*STAR Centre for Frontier AI Research(新加坡科研局前沿人工智能研究中心);ZJU-Angelalign R&D Center for Intelligence Healthcare(浙江大学时代天使智能医疗研发中心)
Transforming the Voice of the Customer: Large Language Models for Identifying Customer Needs
Kellogg School of Management, Northwestern University(西北大学凯洛格管理学院);MIT Management School, Massachusetts Institute of Technology(麻省理工学院斯隆管理学院);Applied Marketing Science, Inc.(应用营销科学公司)
Identifying Semantic Induction Heads to Understand In-Context Learning
Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Catch Me If You Can: Identifying Fraudulent Physician Reviews with Large Language Models Using Generative Pre-Trained Transformers
Beedie School of Business, Simon Fraser University(西蒙菲莎大学比迪商学院);Faculty of Applied Science, Simon Fraser University(西蒙菲莎大学应用科学学院);Carey Business School, John Hopkins University(约翰霍普金斯大学凯瑞商学院)
Assessing the Applicability of Natural Language Processing to Traditional Social Science Methodology: A Case Study in Identifying Strategic Signaling Patterns in Presidential Directives
SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)
University of Wolverhampton(伍尔弗汉普顿大学);Amazon Research(亚马逊研究院);IBM Research(IBM研究院);Columbia University(哥伦比亚大学);Bhim Rao Ambedkar University(比姆·拉奥·安贝德卡尔大学);Qatar Computing Research Institute, HBKU(哈马德·本·哈利法大学卡塔尔计算研究所)
ML_LTU at SemEval-2022 Task 4: T5 Towards Identifying Patronizing and Condescending Language
Luleå University of Technology(吕勒奥理工大学)
本文描述了LTU的机器学习小组在SemEval-2022任务4的子任务1中使用的系统:Patronizing and Condensing Language(PCL)Detection。我们的系统包括微调预训练的文本到文本传输Transformer(T5),并创新地减少其类外预测。本文的主要贡献是:1)描述了我们使用的T5模型的实现细节,2)分析了该模型在这项任务中的成功和困难,3)在正式提交之外进行了消融研究,以确定数据分割的相对重要性。我们的模型在官方测试集上的F1得分为0.5452。