Due to the limitations of the model structure and pre-training objectives, existing vision-and-language generation models cannot utilize pair-wise images and text through bi-directional generation. In this paper, we propose DU-VLG, a framework which unifies vision-and-language generation as sequence generation problems. DU-VLG is trained with novel dual pre-training tasks: multi-modal denoising autoencoder tasks and modality translation tasks. To bridge the gap between image understanding and generation, we further design a novel commitment loss. We compare pre-training objectives on image captioning and text-to-image generation datasets. Results show that DU-VLG yields better performance than variants trained with uni-directional generation objectives or the variant without the commitment loss. We also obtain higher scores compared to previous state-of-the-art systems on three vision-and-language generation tasks. In addition, human judges further confirm that our model generates real and relevant images as well as faithful and informative captions.
Modality-Specialized Synergizers for Interleaved Vision-Language Generalists
Virginia Tech（弗吉尼亚理工大学）；Intuit AI Research（Intuit AI研究院）；Meta AI；The Chinese University of Hong Kong（香港中文大学）；University of California, Davis（加州大学戴维斯分校）
MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations
MedVisAI Lab（MedVisAI实验室）；ECE, Northwestern University（电子工程系，西北大学）；Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息与通信研究所（I2R），A*STAR，新加坡）；Lee Kong Chian School of Medicine, Nanyang Technological University（Lee Kong Chian医学院，南洋理工大学）
Towards Vision-Language Geo-Foundation Model: A Survey
School of GeoAI（地理人工智能学院）；Hindon STAI Institute（Hindon STAI研究所）；Key Laboratory of Geographic Information Science (Ministry of Education)（地理信息科学重点实验室）；East China Normal University（华东师范大学）；School of AI（人工智能学院）；Shanghai Jiao Tong University（上海交通大学）；School of Automation and Intelligent Sensing（自动化与智能感知学院）
FlowInOne:Unifying Multimodal Generation as Image-in, Image-out Flow Matching
University of Electronic Science and Technology of China（电子科技大学）；Central South University（中南大学）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）；Microsoft（微软）
Unifying VLM-Guided Flow Matching and Spectral Anomaly Detection for Interpretable Veterinary Diagnosis
School of Mathematics, Shandong University（山东大学数学学院）；Shenzhen Loop Area Institute（深圳河套学院）；Yeshiva University（叶史瓦大学）；Qilu University of Technology（齐鲁工业大学）；Shandong Normal University（山东师范大学）；Chuzhou University（滁州学院）
Attentive Convolution: Unifying the Expressivity of Self-Attention with Convolutional Efficiency
Center for Machine Vision and Signal Analysis, University of Oulu（机器视觉与信号分析中心，奥卢大学）；Department of Psychiatry and Behavioral Sciences, Stanford University（精神病学与行为科学系，斯坦福大学）；School of Computer Science, Nanjing University of Information Science and Technology（计算机科学学院，南京信息科技大学）；Department of Computer Science, Aalto University（计算机科学系，阿尔托大学）；ELLIS Institute Finland（芬兰ELLIS研究所）
EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
X-Humanoid；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；University of Manchester（曼彻斯特大学）；Monash University（墨尔本大学）；Celonis AI；University of New South Wales（新南威尔士大学）
3D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidance
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Meituan Inc.（美团公司）；University of Trento（特伦托大学）；Peking University（北京大学）；Carnegie Mellon University（卡内基梅隆大学）
Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias
The Ohio State University（俄亥俄州立大学）；Imperial College London（伦敦帝国学院）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Peking University（北京大学）
Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；KEENON Robotics Co., Ltd.（科语机器人有限公司）；Suzhou Silicon Era Intelligent Technology Co., Ltd.（苏州硅时代智能科技有限公司）；College of Materials, Xiamen University（厦门大学材料学院）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；ByteDance（字节跳动）；State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院通用人工智能国家重点实验室）；Hubei Automation Institute（湖北省自动化研究所）
Unifying Vision, Text, and Layout for Universal Document Processing
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Microsoft Azure Cognitive Services Research（微软Azure认知服务研究院）；Microsoft Azure Visual Document Intelligence（微软Azure视觉文档智能部门）
UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs
University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；National University of Singapore（新加坡国立大学）；Zhongguancun Academy（中关村学院）
HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptation
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Alibaba（阿里巴巴）；The Hong Kong University of Science and Technology（香港科技大学）；National University of Singapore（新加坡国立大学）
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation
The Chinese University of Hong Kong（香港中文大学）；Fudan University（复旦大学）；Sensetime Research Institute（商汤研究院）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
UniRS: Unifying Multi-temporal Remote Sensing Tasks through Vision Language Models
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Geographic Sciences, Hunan Normal University（湖南师范大学）；City University of Hong Kong（香港城市大学）
Unifying Adversarially Robust Model Experts in Vision-Language Models
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Center for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局前沿人工智能研究中心）
Unifying Visual and Vision-Language Tracking via Contrastive Learning
Deep Space Exploration Laboratory（深空探测实验室）；University of Science and Technology of China（中国科学技术大学）；Intelligent Science Technology Academy of CASIC（中国航天科工集团智能科学技术研究院）
