Finetuning large pre-trained language models with a task-specific head has advanced the state-of-the-art on many natural language understanding benchmarks. However, models with a task-specific head require a lot of training data, making them susceptible to learning and exploiting dataset-specific superficial cues that do not generalize to other datasets. Prompting has reduced the data requirement by reusing the language model head and formatting the task input to match the pre-training objective. Therefore, it is expected that few-shot prompt-based models do not exploit superficial cues. This paper presents an empirical examination of whether few-shot prompt-based models also exploit superficial cues. Analyzing few-shot prompt-based models on MNLI, SNLI, HANS, and COPA has revealed that prompt-based models also exploit superficial cues. While the models perform well on instances with superficial cues, they often underperform or only marginally outperform random accuracy on instances without superficial cues.
Mitigating the Bias of Large Language Model Evaluation
School of Architecture and Design, Harbin Institute of Technology(哈尔滨工业大学建筑与设计学院);Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);University of Essex(埃塞克斯大学)
In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
Northeastern University(东北大学);Technion -- Israel Institute of Technology(以色列理工学院);Google DeepMind(谷歌DeepMind);New York University(纽约大学);Google Research(谷歌研究院)
Enhancing Low-Resource Relation Representations through Multi-View Decoupling
Cognitive Computing and Intelligent Information Processing (CCIIP) Laboratory, School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院认知计算与智能信息处理实验室);Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL)(华中科技大学与平安财产保险联合实验室);Ping An Property & Casualty Insurance Company of China, Ltd.(中国平安财产保险股份有限公司);The Chinese University of Hong Kong(香港中文大学);Shanghai AI Laboratory(上海人工智能实验室)
Dialect Normalization using Large Language Models and Morphological Rules
Archimedes, Athena Research Center, Greece(阿基米德,雅典娜研究中心,希腊);Department of Informatics and Telecommunications, NKUA(信息与电信系,NKUA);Department of Informatics, Athens University of Economics and Business, Greece(信息系,雅典经济与商业大学,希腊);Department of Computer Science, George Mason University(计算机科学系,乔治·梅森大学)
Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models
Department of Computer Science(计算机科学系);School of Nursing(护理学院);Department of Information Sciences and Technology(信息科学与技术系);Department of Biomedical Informatics & Medical Education(生物医学信息学与医学教育系)
Dissecting Role Cognition in Medical LLMs via Neuronal Ablation
School of Information, Renmin University of China(中国人民大学信息学院);Center of Machine Learning Research, Peking University(北京大学机器学习研究中心);School of Artificial Intelligence, Shanghai Jiaotong University(上海交通大学人工智能学院);Health Informatics Committee of the China Society for Scientific and Technical Information, Renmin University of China(中国科学技术信息学会健康信息技术委员会,中国人民大学);Institute for Advanced Algorithms Research (IAAR), Shanghai, China(先进算法研究所(IAAR),上海,中国);Memtensor Research Center, Shanghai, China(Memtensor研究中心,上海,中国)
Are LLMs Rational Investors? A Study on Detecting and Reducing the Financial Bias in LLMs
School of Computer Science, Fudan University(复旦大学计算机科学技术学院);Institute of Fintech, Fudan University(复旦大学金融科技研究院);School of Electronics and Information Engineering, Tongji University(同济大学电子与信息工程学院);Tandon School of Engineering, New York University(纽约大学坦登工程学院);DataGrand Inc.(数据堂公司)
Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Models
East China Normal University(东华大学);Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);Xi’an Jiaotong University(西安交通大学);Shanghai AI Laboratory(上海人工智能实验室)
Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models
Department of Electrical and Electronic Engineering(电子与电气工程系);The Hong Kong Polytechnic University(香港理工大学);School of Electronics and Information(电子与信息学院);Northwestern Polytechnical University(西北工业大学)
Yet another algorithmic bias: A Discursive Analysis of Large Language Models Reinforcing Dominant Discourses on Gender and Race
Instituto de Estudos da Linguagem (IEL), Universidade Estadual de Campinas (UNICAMP)(语言研究学院(IEL),Campinas州立大学(UNICAMP));Instituto de Computação (IC), Universidade Estadual de Campinas (UNICAMP)(计算学院(IC),Campinas州立大学(UNICAMP));Instituto de Informática, Universidade Federal de Goiás (UFG)(信息学院,戈亚斯联邦大学(UFG))
Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding
Peking University(北京大学);National Key Laboratory for Multimedia Information Processing(多媒体信息处理国家重点实验室);School of Computer Science, Peking University(北京大学计算机学院)
Context-Alignment: Activating and Enhancing LLM Capabilities in Time Series
The Hong Kong Polytechnic University(香港理工大学);Ningbo Institute of Digital Twin(宁波数字孪生研究所);Eastern Institute of Technology(东部技术研究所);Shanghai Jiao Tong University(上海交通大学)