Recently, Convolutional Neural Networks (CNN) based image super-resolution (SR) have shown significant success in the literature. However, these methods are implemented as single-path stream to enrich feature maps from the input for the final prediction, which fail to fully incorporate former low-level features into later high-level features. In this paper, to tackle this problem, we propose a deep interleaved network (DIN) to learn how information at different states should be combined for image SR where shallow information guides deep representative features prediction. Our DIN follows a multi-branch pattern allowing multiple interconnected branches to interleave and fuse at different states. Besides, the asymmetric co-attention (AsyCA) is proposed and attacked to the interleaved nodes to adaptively emphasize informative features from different states and improve the discriminative ability of networks. Extensive experiments demonstrate the superiority of our proposed DIN in comparison with the state-of-the-art SR methods.
Beijing Jiaotong University（北京交通大学）；University of Electronic Science and Technology of China（电子科技大学）
Recently, convolutional neural network (CNN) has demonstrated significant success for image restoration (IR) tasks (e.g., image super-resolution, image deblurring, rain streak removal, and dehazing). However, existing CNN based models are commonly implemented as a single-path stream to enrich feature representations from low-quality (LQ) input space for final predictions, which fail to fully incorporate preceding low-level contexts into later high-level features within networks, thereby producing inferior results. In this paper, we present a deep interleaved network (DIN) that learns how infor…
MIT（麻省理工学院）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）
Fetal motion is unpredictable and rapid on the scale of conventional MR scan times. Therefore, dynamic fetal MRI, which aims at capturing fetal motion and dynamics of fetal function, is limited to fast imaging techniques with compromises in image quality and resolution. Super-resolution for dynamic fetal MRI is still a challenge, especially when multi-oriented stacks of image slices for oversampling are not available and high temporal resolution for recording the dynamics of the fetus or placenta is desired. Further, fetal motion makes it difficult to acquire high-resolution images for supervi…
In this paper, we are interested in building lightweight and efficient convolutional neural networks. Inspired by the success of two design patterns, composition of structured sparse kernels, e.g., interleaved group convolutions (IGC), and composition of low-rank kernels, e.g., bottle-neck modules, we study the combination of such two design patterns, using the composition of structured sparse low-rank kernels, to form a convolutional kernel. Rather than introducing a complementary condition over channels, we introduce a loose complementary condition, which is formulated by imposing the comple…
CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation
The Hong Kong University of Science and Technology（香港科技大学）；Wuhan University（武汉大学）；AI Chip Center for Emerging Smart Systems（新兴智能系统人工智能芯片中心）；Kuaishou Technology（快手科技）
Think in Strokes, Not Pixels: Process-Driven Image Generation via Interleaved Reasoning
Meta Superintelligence Labs（Meta超级智能实验室）；University of California, San Diego（加州大学圣迭戈分校）；Worcester Polytechnic Institute（伍斯特理工学院）；Northwestern University（西北大学）
Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiaotong University（上海交通大学）；Zhejiang University（浙江大学）；University of Chinese Academy of Sciences（中国科学院大学）
InterCoG: Towards Spatially Precise Image Editing with Interleaved Chain-of-Grounding Reasoning
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Zhengzhou Advanced Research Institute of Harbin Institute of Technology（哈尔滨工业大学郑州先进研究院）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳先进技术大学人工智能研究院）
Interleaved Latent Visual Reasoning with Selective Perceptual Modeling
China University of Geosciences, Wuhan（中国地质大学（武汉））；Shanghai Innovation Institute（上海创新研究院）；University of Southern California（南加州大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；Shanghai Jiao Tong University（上海交通大学）
We introduce Emu3.5, a large-scale multimodal world model that natively predicts the next state across vision and language. Emu3.5 is pre-trained end-to-end with a unified next-token prediction objective on a corpus of vision-language interleaved data containing over 10 trillion tokens, primarily derived from sequential frames and transcripts of internet videos. The model naturally accepts interleaved vision-language inputs and generates interleaved vision-language outputs. Emu3.5 is further post-trained with large-scale reinforcement learning to enhance multimodal reasoning and generation. To…
National Institutes of Health Clinical Center（美国国立卫生研究院临床中心）
Despite tremendous progress in computer vision, there has not been an attempt for machine learning on very large-scale medical image databases. We present an interleaved text/image deep learning system to extract and mine the semantic interactions of radiology images and reports from a national research hospital's Picture Archiving and Communication System. With natural language processing, we mine a collection of representative ~216K two-dimensional key images selected by clinicians for diagnostic reference, and match the images with their descriptions in an automated manner. Our system inter…
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text
University of California, Santa Barbara（加州大学圣塔芭芭拉分校）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗·G·艾伦计算机科学学院）；Columbia University（哥伦比亚大学）；Yonsei University（延世大学）；LAION
