Samuel Weinbach, Marco Bellagente, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Björn Deiseroth, Koen Oostermeijer, Hannah Teufel, Andres Felipe Cruz-Salinas
We introduce M-VADER: a diffusion model (DM) for image generation where the output can be specified using arbitrary combinations of images and text. We show how M-VADER enables the generation of images specified using combinations of image and text, and combinations of multiple images. Previously, a number of successful DM image generation algorithms have been introduced that make it possible to specify the output image using a text prompt. Inspired by the success of those models, and led by the notion that language was already developed to describe the elements of visual contexts that humans find most important, we introduce an embedding model closely related to a vision-language model. Specifically, we introduce the embedding model S-MAGMA: a 13 billion parameter multimodal decoder combining components from an autoregressive vision-language model MAGMA and biases finetuned for semantic search.
Consolidating Diffusion-Generated Video Detection with Unified Multimodal Forgery Learning
School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机科学学院);School of Information Science and Electronic Engineering, Shanghai Jiao Tong University(上海交通大学信息科学与电子工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Department of Computer Science and Engineering, Michigan State University(密歇根州立大学计算机科学与工程系)
Novel Diffusion Models for Multimodal 3D Hand Trajectory Prediction
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University(IRMV实验室,自动化系,上海交通大学);Meta Reality Labs(Meta现实实验室);the Department of Electronic Engineering, Shanghai Jiao Tong University(电子工程系,上海交通大学);the School of Information and Control Engineering, China University of Mining and Technology(信息与控制工程学院,中国矿业大学);the College of Intelligence Science and Technology, National University of Defense Technology(智能科学与技术学院,国防科技大学)
Multimodal-Conditioned Latent Diffusion Models for Fashion Image Editing
Media Integration and Communication Center (MICC), University of Florence(佛罗伦萨大学媒体集成与传播中心(MICC));University of Florence(佛罗伦萨大学);University of Modena and Reggio Emilia(摩德纳-雷焦艾米利亚大学);University of Pisa(比萨大学)
Diagnosing and Correcting Concept Omission in Multimodal Diffusion Transformers
Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea;Department of Electrical;Computer Engineering, Seoul National University, Seoul, South Korea;Department of Computer Science \& Engineering, Korea University, Seoul, South Korea;ISRC, Seoul National University, Seoul, South Korea
Intention Enhanced Diffusion Model for Multimodal Pedestrian Trajectory Prediction
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, the Southern University of Science and Technology, Shenzhen 518055, China(广东省全自动化系统控制理论与技术重点实验室,南方科技大学,深圳518055,中国);Department of Mechanical Engineering, City University of Hong Kong, Hong Kong SAR, China(香港城市大学机械工程系,香港特别行政区,中国)
Adversarial-Guided Diffusion for Multimodal LLM Attacks
School of Information Science and Engineering, Lanzhou University(兰州大学信息科学与工程学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
Rethinking Cross-Modal Interaction in Multimodal Diffusion Transformers
The University of Hong Kong(香港大学);Nanjing University(南京大学);University of Chinese Academy of Sciences(中国科学院大学);Nanyang Technological University(南洋理工大学);Harbin Institute of Technology(哈尔滨工业大学)
Diff4MMLiTS: Advanced Multimodal Liver Tumor Segmentation via Diffusion-Based Image Synthesis and Alignment
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China(电子与电气工程系,南方科技大学,深圳,中国);Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong SAR, China(电气与电子工程系,香港大学,香港特别行政区,中国);Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing, China(放射科,中大医院,医学院,东南大学,南京,中国);Jiaxing Research Institute, Southern University of Science and Technology, Jiaxing, China(嘉兴研究所,南方科技大学,嘉兴,中国)
MMA-Diffusion: MultiModal Attack on Diffusion Models
The Chinese University of Hong Kong(香港中文大学);Huawei(华为);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Wenge Technology Co. Ltd(北京文格科技有限公司)
DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing
Shenzhen Technology University(深圳科技大学);University of Washington(华盛顿大学);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院)
OSDM-MReg: Multimodal Image Registration based One Step Diffusion Model
Shanghai Key Laboratory of Intelligent Sensing and Recognition(上海智能感知与识别重点实验室);Shanghai Jiao Tong University(上海交通大学);Center of Digital Innovation(数字创新中心)