There is growing interest in integrating high-fidelity visual synthesis capabilities into large language models (LLMs) without compromising their strong reasoning capabilities. Existing methods that directly train LLMs or bridge LLMs and diffusion models usually suffer from costly training since the backbone LLMs have not seen image representations during pretraining. We present Bifrost-1, a unified framework that bridges pretrained multimodal LLMs (MLLMs) and diffusion models using patch-level CLIP image embeddings as latent variables, which are natively aligned with the MLLM's CLIP visual encoder. These patch-level image embeddings are integrated into the diffusion model with a lightweight adaptation of its ControlNet. To retain the original multimodal reasoning capabilities of MLLMs, we equip the MLLM with a visual generation branch initialized from the original MLLM parameters when predicting the patch-level image embeddings. By seamlessly integrating pretrained MLLMs and diffusion models with patch-level CLIP latents, our framework enables high-fidelity controllable image generation with significant training efficiency. Our experiments demonstrate that Bifrost-1 achieves comparable or better performance than previous methods in terms of visual fidelity and multimodal understanding, with substantially lower compute during training. We also provide comprehensive ablation studies showing the effectiveness of our design choices.
Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；Institute of Medical Intelligence and XR, The Chinese University of Hong Kong（香港中文大学医学智能与扩展现实研究所）
Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
Department of Electrical and Computer Engineering, Seoul National University（首尔国立大学电子与计算机工程系）；Amazon（亚马逊）；Samsung Research（三星研究院）；School of Computer Science and Engineering, Soongsil University（顺天大学计算机科学与工程学院）；IPAI, AIIS, ASRI, INMC, and ISRC, Seoul National University（首尔国立大学IPAI、AIIS、ASRI、INMC和ISRC）
Bridging Weakly-Supervised Learning and VLM Distillation: Noisy Partial Label Learning for Efficient Downstream Adaptation
Institute of Perceptual Intelligence, Peng Cheng Laboratory（感知智能研究院，鹏城实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）
Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
King Abdullah University of Science and Technology（国王阿卜杜勒-阿齐兹大学科学与技术大学）；Visual Recognition Group（视觉识别组）；Faculty of Electrical Engineering（电气工程学院）；Czech Technical University in Prague（布拉格捷克技术大学）
VidBridge-R1: Bridging QA and Captioning for RL-based Video Understanding Models with Intermediate Proxy Tasks
New Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所模式识别实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Kling Team, Kuaishou Technology（快手科技 Kling 团队）；Peking University（北京大学）；Nanjing University（南京大学）
Bridging Text and Image for Artist Style Transfer via Contrastive Learning
Lappeenranta-Lahti University of Technology LUT（拉彭兰塔-拉赫蒂理工大学（LUT））；The Hong Kong Polytechnic University（香港理工大学）；Ecole Polytechnique（巴黎综合理工学院）；IP Paris（巴黎文理大学）
