Having revolutionized natural language processing (NLP) applications, large language models (LLMs) are expanding into the realm of multimodal inputs. Owing to their ability to interpret images, multimodal LLMs (MLLMs) have been primarily used for vision-language tasks. Currently, MLLMs have not yet been extended for domain-specific visual tasks, which require a more explicit understanding of visual information. We developed a method to transform domain-specific visual and vision-language datasets into a unified question answering format called Visual Question Answering Instruction (VQA-IN), thereby extending MLLM to domain-specific tasks. The VQA-IN was applied to train multiple MLLM architectures using smaller versions of LLMs (sLLMs). The experimental results indicated that the proposed method achieved a high score metric on domainspecific visual tasks while also maintaining its performance on vision-language tasks in a multitask manner.
Visual Question Answering (VQA) concerns providing answers to Natural Language questions about images. Several deep neural network approaches have been proposed to model the task in an end-to-end fashion. Whereas the task is grounded in visual processing, if the question focuses on events described by verbs, the language understanding component becomes crucial. Our hypothesis is that models should be aware of verb semantics, as expressed via semantic role labels, argument types, and/or frame elements. Unfortunately, no VQA dataset exists that includes verb semantic information. Our first contr…
Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning
Wuhan University（武汉大学）；Xiaomi Inc（小米公司）；Wuhan University of Technology（武汉理工大学）；Nanyang Technological University（南洋理工大学）；The Hong Kong University of Science and Technology（香港科技大学）
A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation
University of Florida（佛罗里达大学）；University of California, San Francisco（加利福尼亚大学旧金山分校）；Stony Brook University（纽约州立大学石溪分校）；The University of Texas MD Anderson Cancer Center（得克萨斯大学MD安德森癌症中心）
University of Helsinki（赫尔辛基大学）；Artificial Intelligence Research Institute（人工智能研究院）
Supporting the current trend in the AI community, we present the AI Journey 2021 Challenge called Fusion Brain, the first competition which is targeted to make the universal architecture which could process different modalities (in this case, images, texts, and code) and solve multiple tasks for vision and language. The Fusion Brain Challenge combines the following specific tasks: Code2code Translation, Handwritten Text recognition, Zero-shot Object Detection, and Visual Question Answering. We have created datasets for each task to test the participants' submissions on it. Moreover, we have co…
While few-shot learning as a transfer learning paradigm has gained significant traction for scenarios with limited data, it has primarily been explored in the context of building unimodal and unilingual models. Furthermore, a significant part of the existing literature in the domain of few-shot multitask learning perform in-context learning which requires manually generated prompts as the input, yielding varying outcomes depending on the level of manual prompt-engineering. In addition, in-context learning suffers from substantial computational, memory, and storage costs which eventually leads …
MedViLaM: A multimodal large language model with advanced generalizability and explainability for medical data understanding and generation
Centre for Perceptual and Interactive Intelligence, the Chinese University of Hong Kong（香港中文大学感知与交互智能中心）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Department of Electronic Engineering, the Chinese University of Hong Kong（香港中文大学电子工程系）
GeoChat: Grounded Large Vision-Language Model for Remote Sensing
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Birla Institute of Technology & Science, Hyderabad（海得拉巴贝拉理工学院）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
Detecting and Evaluating Medical Hallucinations in Large Vision Language Models
Academy for Engineering and Technology, Fudan University（复旦大学工程与技术学院）；Tencent Youtu Lab（腾讯优图实验室）；Cognition and Intelligent Technology Laboratory（认知与智能技术实验室）
PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue
Paige, NYC, NY United States（美国纽约市帕伊公司）；Microsoft Research, Cambridge, MA United States（微软研究院）；Memorial Sloan Kettering Cancer Center, NYC, NY United States（纪念斯隆凯特林癌症中心）；University of Yale, New Haven, CT United States（耶鲁大学）
A Versatile Pathology Co-pilot via Reasoning Enhanced Multimodal Large Language Model
Hong Kong University of Science and Technology（香港科技大学）；Southern Medical University（南方医科大学）；Nanfang Hospital and School of Basic Medical Sciences（南方医院和基础医学科学学院）；Chinese University of Hong Kong（香港中文大学）
Specialty-Oriented Generalist Medical AI for Chest CT Screening
Rensselaer Polytechnic Institute（伦斯勒理工学院）；Wake Forest University School of Medicine（维克森林大学医学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
