Finding the best way of adapting pre-trained language models to a task is a big challenge in current NLP. Just like the previous generation of task-tuned models (TT), models that are adapted to tasks via in-context-learning (ICL) are robust in some setups but not in others. Here, we present a detailed analysis of which design choices cause instabilities and inconsistencies in LLM predictions. First, we show how spurious correlations between input distributions and labels -- a known issue in TT models -- form only a minor problem for prompted models. Then, we engage in a systematic, holistic evaluation of different factors that have been found to influence predictions in a prompting setup. We test all possible combinations of a range of factors on both vanilla and instruction-tuned (IT) LLMs of different scale and statistically analyse the results to show which factors are the most influential, interactive or stable. Our results show which factors can be used without precautions and which should be avoided or handled with care in most settings.
Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models
School of Computing University of Kent(肯特大学计算机学院);Leiden Institute of Advanced Computer Science Universiteit Leiden(莱顿先进计算机科学研究所);Cheriton School of Computer Science University of Waterloo(滑铁卢大学查里顿计算机科学学院)
From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University(上海智能信息处理关键实验室,复旦大学计算机学院);Shanghai Collaborative Innovation Center on Intelligent Visual Computing(上海智能视觉计算协同创新中心);Singapore Management University(新加坡管理大学)
MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学)
The Cultural Psychology of Large Language Models: Is ChatGPT a Holistic or Analytic Thinker?
New York University(纽约大学);University of Chicago(芝加哥大学);Massachusetts Institute of Technology(麻省理工学院);Johns Hopkins University(约翰斯·霍普金斯大学);University of California Los Angeles(加州大学洛杉矶分校)
Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoning
University of California, Santa Barbara(加州大学圣巴巴拉分校);The University of Texas at Dallas(德克萨斯大学达拉斯分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校);University of Maryland, College Park(马里兰大学帕克分校)
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
DAMO Academy, Alibaba Group(达摩院,阿里巴巴集团);Hupan Laboratory(湖畔实验室);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Department of Radiology, The Affiliated Yangming Hospital of Ningbo University(宁波大学附属阳明医院放射科);Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University(浙江大学伊利诺伊大学厄巴纳香槟校区联合学院,浙江大学);Hepato-Pancreato-Biliary Center, Beijing Tsinghua Changgung Hospital, School of Clinical Medicine, Tsinghua Medicine, Tsinghua University(清华长庚医院肝胆胰中心,清华大学临床医学院,清华医学,清华大学);School of Software, Tsinghua University(清华大学软件学院);Beijing National Research Center for Information Science and Technology, Tsinghua University(清华大学北京信息科学与技术国家研究中心)
Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion
NeuBCI Lab, State Key Laboratory of Brain Cognition;Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences, Beijing, China;School of Future Technology, University of Chinese Academy of Sciences, Beijing, China;School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China;Zhongguancun Academy, Beijing, China;Peking University, Beijing, China
Mind the Gap: A Review of Arabic Post-Training Datasets and Their Limitations
Refine AI;ASAS AI;University of Bisha(比沙大学);University College London(伦敦大学学院);King Salman Global Academy for Arabic(萨勒曼全球阿拉伯学院);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);King Abdulaziz University(阿卜杜勒阿齐兹大学);HUMAIN
本文综述 Hugging Face Hub 上的阿拉伯语后训练数据集,从能力、可引导性、对齐和鲁棒性等维度评估其质量与局限,并提出未来建设建议。
Mind the Rarities: Can Rare Skin Diseases Be Reliably Diagnosed via Diagnostic Reasoning?
Carnegie Mellon University(卡内基梅隆大学);University of Alabama at Birmingham(阿拉巴马大学伯明翰分校);Australian National University(澳大利亚国立大学);University of Western Australia(西澳大学);POSTECH;Yale University(耶鲁大学)
Falcon-UI: Understanding GUI Before Following User Instructions
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);College of Computer Science, Nankai University(南开大学计算机学院);Department of Automation, Tsinghua University(清华大学自动化系);Beijing Academy of Artificial Intelligence(北京人工智能研究院);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
GIEBench: Towards Holistic Evaluation of Group Identity-based Empathy for Large Language Models
Beijing University of Posts and Telecommunications(北京邮电大学);Tianjin University(天津大学);M-A-P;Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Waterloo(滑铁卢大学)
ASMA-Tune: Unlocking LLMs' Assembly Code Comprehension via Structural-Semantic Instruction Tuning
Nankai University(南开大学);Zhejiang University(浙江大学);Ant Group(蚂蚁集团);Renmin University of China(中国人民大学);University of the Chinese Academy of Sciences(中国科学院大学)