Recently, generative domain adaptation has achieved remarkable progress, enabling us to adapt a pre-trained generator to a new target domain. However, existing methods simply adapt the generator to a single target domain and are limited to a single modality, either text-driven or image-driven. Moreover, they cannot maintain well consistency with the source domain, which impedes the inheritance of the diversity. In this paper, we propose UniHDA, a \textbf{unified} and \textbf{versatile} framework for generative hybrid domain adaptation with multi-modal references from multiple domains. We use CLIP encoder to project multi-modal references into a unified embedding space and then linearly interpolate the direction vectors from multiple target domains to achieve hybrid domain adaptation. To ensure \textbf{consistency} with the source domain, we propose a novel cross-domain spatial structure (CSS) loss that maintains detailed spatial structure information between source and target generator. Experiments show that the adapted generator can synthesise realistic images with various attribute compositions. Additionally, our framework is generator-agnostic and versatile to multiple generators, e.g., StyleGAN, EG3D, and Diffusion Models.
Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Human-centric perception plays a vital role in vision and graphics. But their data annotations are prohibitively expensive. Therefore, it is desirable to have a versatile pre-train model that serves as a foundation for data-efficient downstream tasks transfer. To this end, we propose the Human-Centric Multi-Modal Contrastive Learning framework HCMoCo that leverages the multi-modal nature of human data (e.g. RGB, depth, 2D keypoints) for effective representation learning. The objective comes with two main challenges: dense pre-train for multi-modality data, efficient usage of sparse human prior…
MASSTAR: A Multi-Modal and Large-Scale Scene Dataset with a Versatile Toolchain for Surface Prediction and Completion
School of Artificial Intelligence, Sun Yat-Sen University（中山大学人工智能学院）；School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；School of Future Technology, Harbin Institute of Technology（哈尔滨工业大学未来技术学院）；Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology（香港科技大学电子与计算机工程系）
OralGPT-Omni: A Versatile Dental Multimodal Large Language Model
Faculty of Dentistry, The University of Hong Kong（香港大学牙科学院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机科学与软件工程学院）；The Hong Kong University of Science and Technology (GZ)（香港科学与技术大学）；School of Biomedical Engineering, Southern Medical University（南方医科大学生物医学工程学院）；Singapore University of Technology and Design（新加坡科技与设计大学）；University of Auckland（奥克兰大学）；University of Science and Technology of China（中国科学技术大学）；School of Computer Science, Peking University（北京大学计算机学院）；College of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）
DeepMind（深度思维）；University of Geneva（日内瓦大学）；HES-SO（瑞士西部应用科学与艺术大学）；Geneva School of Business Administration（日内瓦工商管理学院）；University of Oxford（牛津大学）
Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -- a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a comm…
HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modeling
Shanghai AI Laboratory（上海人工智能实验室）；S-Lab；Nanyang Technological University（南洋理工大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）
Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOti…
LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering
OpenGVLab, Shanghai AI Laboratory, China（OpenGVLab，上海人工智能实验室）；University of Science and Technology of China（中国科学技术大学）；Honor Device Co.,Ltd（荣耀设备有限公司）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究所，中国科学院）；Nanjing University（南京大学）
With the rise and development of deep learning over the past decade, there has been a steady momentum of innovation and breakthroughs that convincingly push the state-of-the-art of cross-modal analytics between vision and language in multimedia field. Nevertheless, there has not been an open-source codebase in support of training and deploying numerous neural network models for cross-modal analytics in a unified and modular fashion. In this work, we propose X-modaler -- a versatile and high-performance codebase that encapsulates the state-of-the-art cross-modal analytics into several general-p…
User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language. The tasks are also diverse, including object detection, language generation and grounding. In this paper, we present VUT, a Versatile UI Transformer that takes multimodal input and simultaneously accomplishes 5 distinct tasks with the same model. Our model consists of a multimodal Transformer encoder that jointly encodes UI images and structures, and performs UI object detection when the UI structures are absent in the input. Our model also consists of an auto-regre…
Find Them All: Unveiling MLLMs for Versatile Person Re-identification
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Institute of Image Communication and Information Processing, Shanghai Jiao Tong University（上海交通大学图像通信与信息处理研究所）；Macao Polytechnic University（澳门 polytechnic 大学）；Shanghai AI Laboratory（上海人工智能实验室）
Beijing Institute of Technology（北京理工大学）；ETH Zurich（苏黎世联邦理工学院）；University of Technology Sydney（悉尼科技大学）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）
With the emergence of varied visual navigation tasks (e.g, image-/object-/audio-goal and vision-language navigation) that specify the target in different ways, the community has made appealing advances in training specialized agents capable of handling individual navigation tasks well. Given plenty of embodied navigation tasks and task-specific solutions, we address a more fundamental question: can we learn a single powerful agent that masters not one but multiple navigation tasks concurrently? First, we propose VXN, a large-scale 3D dataset that instantiates four classic navigation tasks in s…
National University of Singapore（新加坡国立大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Shandong University（山东大学）；Shandong Artificial Intelligence Institute（山东人工智能研究院）；University of Florence（佛罗伦萨大学）
Making each modality in multi-modal data contribute is of vital importance to learning a versatile multi-modal model. Existing methods, however, are often dominated by one or few of modalities during model training, resulting in sub-optimal performance. In this paper, we refer to this problem as modality bias and attempt to study it in the context of multi-modal classification systematically and comprehensively. After stepping into several empirical analysis, we recognize that one modality affects the model prediction more just because this modality has a spurious correlation with instance lab…
Nanyang Technological University（南洋理工大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；SenseTime Research（商汤科技研究院）
We introduce a simple and versatile framework for image-to-image translation. We unearth the importance of normalization layers, and provide a carefully designed two-stream generative model with newly proposed feature transformations in a coarse-to-fine fashion. This allows multi-scale semantic structure information and style representation to be effectively captured and fused by the network, permitting our method to scale to various tasks in both unsupervised and supervised settings. No additional constraints (e.g., cycle consistency) are needed, contributing to a very clean and simple method…
M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；Southern University of Science and Technology（南方科技大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Tencent Youtu Lab（腾讯优图实验室）；University of the Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）
EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain
Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology（北京理工大学多学科科学先进研究院）；School of Mechatronical Engineering, Beijing Institute of Technology（北京理工大学机电工程学院）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）；Beijing Institute of Technology（北京理工大学）
MMLF: Multi-modal Multi-class Late Fusion for Object Detection with Uncertainty Estimation
Glasgow College, University of Electronic Science and Technology of China（电子科技大学格拉斯哥学院）；School of Automation Engineering, University of Electronic Science and Technology of China（电子科技大学自动化工程学院）
As cameras and inertial sensors are becoming ubiquitous in mobile devices and robots, it holds great potential to design visual-inertial navigation systems (VINS) for efficient versatile 3D motion tracking which utilize any (multiple) available cameras and inertial measurement units (IMUs) and are resilient to sensor failures or measurement depletion. To this end, rather than the standard VINS paradigm using a minimal sensing suite of a single camera and IMU, in this paper we design a real-time consistent multi-IMU multi-camera (MIMC)-VINS estimator that is able to seamlessly fuse multi-modal …
Enhancing CT Image synthesis from multi-modal MRI data based on a multi-task neural network framework
Kennedy Krieger Institute（肯尼迪克里格研究所）；Columbia University（哥伦比亚大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
FoMo: Multi-Modal, Multi-Scale and Multi-Task Remote Sensing Foundation Models for Forest Monitoring
National Observatory of Athens（雅典国家天文台）；National Technical University of Athens（雅典国家技术大学）；Harokopio University of Athens（哈罗科皮奥大学）；McGill University（麦吉尔大学）
Cascaded Multi-Modal Mixing Transformers for Alzheimer's Disease Classification with Incomplete Data
Queensland Brain Institute, The University of Queensland（昆士兰脑研究所，昆士兰大学）；School of Information Technology and Electrical Engineering, The University of Queensland（信息技术与电气工程学院，昆士兰大学）
The Hong Kong University of Science and Technology（香港科技大学）；ByteDance AI Lab（字节跳动人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）
Recent advances in vision-language pre-training have pushed the state-of-the-art on various vision-language tasks, making machines more capable of multi-modal writing (image-to-text generation) and painting (text-to-image generation). However, few studies investigate if these two essential capabilities can be learned together and boost each other, making a versatile and powerful multi-modal foundation model. In this work, we disclose the potential of symmetric generative vision-language pre-training in learning to write and paint concurrently, and propose a new unified modal model, named DaVin…
The University of Hong Kong（香港大学）；Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）深圳大数据研究院）；Longgang Central Hospital of Shenzhen（深圳龙岗中心医院）；Longgang People Hospital of Shenzhen（深圳龙岗人民医院）
Despite the considerable progress in automatic abdominal multi-organ segmentation from CT/MRI scans in recent years, a comprehensive evaluation of the models' capabilities is hampered by the lack of a large-scale benchmark from diverse clinical scenarios. Constraint by the high cost of collecting and labeling 3D medical data, most of the deep learning models to date are driven by datasets with a limited number of organs of interest or samples, which still limits the power of modern deep models and makes it difficult to provide a fully comprehensive and fair estimate of various methods. To miti…
Cross-modal learning for SAR target recognition using optical vision foundation models
School of Engineering, University of Edinburgh（爱丁堡大学工程学院）；Leonardo UK（英国莱昂纳多公司）；School of Engineering and Physical Sciences, Heriot-Watt University（赫瑞瓦特大学工程与物理科学学院）
