Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learning
Tsinghua University(清华大学);OpenGVLab(开放视觉实验室);Shanghai AI Laboratory(上海人工智能实验室);SenseTime Research(商汤科技研究院);The Chinese University of Hong Kong(香港中文大学);University of Science and Technology of China(中国科学技术大学);Xi’an Jiaotong University(西安交通大学);Beijing University of Posts and Telecommunications(北京邮电大学)
Chenyu Yang, Xizhou Zhu, Jinguo Zhu, Weijie Su, Junjie Wang, Xuan Dong, Wenhai Wang, Lewei Lu, Bin Li, Jie Zhou, Yu Qiao, Jifeng Dai
Recently, vision model pre-training has evolved from relying on manually annotated datasets to leveraging large-scale, web-crawled image-text data. Despite these advances, there is no pre-training method that effectively exploits the interleaved image-text data, which is very prevalent on the Internet. Inspired by the recent success of compression learning in natural language processing, we propose a novel vision model pre-training method called Latent Compression Learning (LCL) for interleaved image-text data. This method performs latent compression learning by maximizing the mutual information between the inputs and outputs of a causal attention model. The training objective can be decomposed into two basic tasks: 1) contrastive learning between visual representation and preceding context, and 2) generating subsequent text based on visual representation. Our experiments demonstrate that our method not only matches the performance of CLIP on paired pre-training datasets (e.g., LAION), but can also leverage interleaved pre-training data (e.g., MMC4) to learn robust visual representation from scratch, showcasing the potential of vision model pre-training with interleaved image-text data. Code is released at https://github.com/OpenGVLab/LCL.
Imagine Before You Predict: Interleaved Latent Visual Reasoning for Video Event Prediction
University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);City University of Hong Kong(香港城市大学);Nanjing University(南京大学);Fudan University(复旦大学);Zhejiang University(浙江大学);University of Electronic Science and Technology of China(电子科技大学)
Interleaved Latent Visual Reasoning with Selective Perceptual Modeling
China University of Geosciences, Wuhan(中国地质大学(武汉));Shanghai Innovation Institute(上海创新研究院);University of Southern California(南加州大学);Fudan University(复旦大学);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学)
Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space
The Hong Kong Polytechnic University(香港理工大学);Singapore Management University(新加坡管理学院);Shandong University(山东大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Interleaved Block-based Learned Image Compression with Feature Enhancement and Quantization Error Compensation
School of software, Shandong University(山东大学软件学院);School of Control Science and Engineering, Shandong University(山东大学控制科学与工程学院);School of Engineering and Sustainable Development, De Montfort University(德蒙福特大学工程与可持续发展学院);College of Computer Science and Software Engineering, Shenzhen University(深圳大学计算机与软件学院);School of Telecommunications Engineering, Xidian University(西安电子科技大学通信工程学院)
MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(脑启发式智能感知与认知联合实验室,中国科学技术大学);University of Missouri, Kansas City(密苏里大学,堪萨斯城)
Generative Video Compression with One-Dimensional Latent Representation
University of Science and Technology of China(中国科学技术大学);Communication University of China(通信大学);Microsoft Research Asia(微软亚洲研究院);University of Newcastle(新castle大学)
UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);Zhongguancun Academy(中关村学院)
Dual-Latent Collaborative Decoding for Fidelity-Perception Balanced Image Compression
School of Information and Communication Engineering and the State Key Laboratory of Media Convergence and Communication, Communication University of China(信息与通信工程学院和媒体融合与通信国家重点实验室,中国通信大学);School of Information Science and Electronic Engineering, Shanghai Jiao Tong University(信息科学与电子工程学院,上海交通大学);Department of Computer Science, City University of Hong Kong(计算机科学系,香港城市大学);State Key Laboratory of Multimedia information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学)
Generative Neural Video Compression via Video Diffusion Prior
School of Information and Communication Engineering, Communication University of China(信息与通信工程学院,通信大学);School of Computer Science, Peking University(计算机学院,北京大学)
Progressive Neural Image Compression with Nested Quantization and Latent Ordering
Qualcomm AI Research, Qualcomm Technologies, Inc.(高通技术公司高通人工智能研究院);University of California Irvine(加州大学欧文分校);Qualcomm AI Research, Qualcomm Technologies Netherlands B.V.(高通技术荷兰有限公司高通人工智能研究院)
LVPNet: A Latent-variable-based Prediction-driven End-to-end Framework for Lossless Compression of Medical Images
Harbin Institute of Technology, China(哈尔滨工业大学);Nanjing University of Information Science and Technology, China(南京信息工程大学);Dalian University of Technology, China(大连理工大学);Nanyang Technological University, Singapore(南洋理工大学)
Compact Latent Representation for Image Compression (CLRIC)
Fraunhofer Institute for Integrated Circuits IIS(弗劳恩霍夫集成电路研究所);Department of Physics, Faculty of Science, Sohag University(科学学院物理系,索哈格大学);Friedrich-Alexander University at Erlangen-Nürnberg(埃尔兰根-纽伦堡弗里德里希-亚历山大大学)
本文提出 CLRIC,利用预训练模型潜在变量和过拟合可学习函数实现任意质量级别的感知图像压缩,兼具低复杂度、模型与分辨率无关性及接近 SOTA 的质量。
Group-of-Latents: Perceptual Video Compression at Extreme Bitrates via Masked Latent Generative Modeling
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China(多媒体信息处理国家重点实验室,计算机科学学院,北京大学,北京,中国);State Key Discipline Laboratory of Wide Band-Gap Semiconductor Technology, School of Microelectronics, Xidian University, Xi'an, China(宽带隙半导体技术重点学科实验室,微电子学院,西安电子科技大学,西安,中国)