Following the impressive development of LLMs, vision-language alignment in LLMs is actively being researched to enable multimodal reasoning and visual IO. This direction of research is particularly relevant to medical imaging because medical image analysis and generation consist of reasoning based on a combination of visual features and prior knowledge. Many recent works have focused on training adapter networks that serve as an information bridge between image processing networks and LLMs; but presumably, in order to achieve maximum reasoning potential of LLMs on visual information as well, visual and language features should be allowed to interact more freely. This is especially important in the medical domain because understanding and generating medical images such as chest X-rays (CXR) require not only accurate visual and language-based reasoning but also a more intimate mapping between the two modalities. Thus, taking inspiration from previous work on the transformer and VQ-GAN combination for bidirectional image and text generation, we build upon this approach and develop a method for instruction-tuning an LLM pre-trained only on text to gain vision-language capabilities for medical images. Specifically, we leverage a pretrained LLM's existing question-answering and instruction-following abilities to teach it to understand visual inputs by instructing it to answer questions about image inputs and, symmetrically, output both text and image responses appropriate to a given query by tuning the LLM with diverse tasks that encompass image-based text-generation and text-based image-generation. We show that our model, LLM-CXR, trained in this approach shows better image-text alignment in both CXR understanding and generation tasks while being smaller in size compared to previously developed models that perform a narrower range of tasks. The code is at https://github.com/hyn2028/llm-cxr.
A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Tasks
LIA, Avignon Université(阿维尼翁大学 LIA 实验室);Zenidoc(Zenidoc 公司);Nantes Université(南特大学);École Centrale Nantes(南特中央理工学院);CNRS(法国国家科学研究中心);LS2N, UMR 6004(南特数字科学实验室)
RadVLM: A Multitask Conversational Vision-Language Model for Radiology
Department of Radiology, Kobe University(金泽大学放射科);Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);Department of Advanced Imaging in Medical Magnetic Resonance, Kyoto University(京都大学医学磁共振高级成像部门);Department of Quantitative Biomedicine, University of Zurich(苏黎世大学定量生物医学系);Diagnostic and Interventional Radiology, University Hospital Zurich(苏黎世大学医院诊断与介入放射科)
SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
Allen Institute for AI(艾伦人工智能研究所);Yale University(耶鲁大学);University of Washington(华盛顿大学);Hebrew University(希伯来大学);MIT(麻省理工学院);Northwestern University(西北大学)
CX-Mind: A Pioneering Multimodal Large Language Model for Interleaved Reasoning in Chest X-ray via Curriculum-Guided Reinforcement Learning
College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine(上海交通大学医学院健康科学与技术学院);Shanghai Innovation Institute(上海创新研究院);Clinical Center for Sports Medicine, Department of Orthopaedics, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine(上海交通大学医学院骨科临床中心);School of Basic Medical Sciences, Intelligent Medicine Institute, Fudan University(复旦大学基础医学学院);Department of Hematology, The First Affiliated Hospital, College of Medicine, Zhejiang University(浙江大学医学院第一附属医院血液科);MoE Key Laboratory of Brain-Inspired Intelligent Perception and Cognition, University of Science and Technology of China(中国科学技术大学脑启发智能感知与认知教育部重点实验室);Department of Public Health and Primary Care, University of Cambridge(剑桥大学公共卫生与初级保健学院);Department of Medicine, Faculty of Health Sciences, Universidad CEU Cardenal Herrera(CEU卡德纳尔-赫尔曼大学健康科学学院医学系);Faculty of Medicine, University of Helsinki(赫尔辛基大学医学院);X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院X-LANCE实验室);Department of Hepatobiliary Surgery, National Cancer Center / National Clinical Research Center for Cancer / Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College(中国医学科学院肿瘤医院肝胆外科);Department of Surgery, The Ohio State University Wexner Medical Center, The James Comprehensive Cancer Center(俄亥俄州立大学韦克斯纳医学中心外科部,詹姆斯综合癌症中心);Ningbo Institute of Technology, Beihang University(北航宁波理工学院)
Self-Instruct: Aligning Language Models with Self-Generated Instructions
University of Washington(华盛顿大学);Tehran Polytechnic(德黑兰理工大学);Arizona State University(亚利桑那州立大学);Johns Hopkins University(约翰斯·霍普金斯大学);Allen Institute for AI(艾伦人工智能研究所)
SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks
Stony Brook University(石溪大学);JP Morgan Chase(摩根大通);Bloomberg(彭博公司);Preferabli(普雷费布利公司);University of Pennsylvania(宾夕法尼亚大学);National Institute of Health(美国国立卫生研究院);Grammarly(grammarly公司)
Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuning
China University of Petroleum (East China)(中国石油大学(华东));The University of Sydney(悉尼大学);JD Explore Academy(京东探索研究院);Nanyang Technological University(南洋理工大学)