We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of 768 * 1360 pixels. Previous video generation models often had limited movement and short durations, and is difficult to generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we propose a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions, to improve both compression rate and video fidelity. Second, to improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing a progressive training and multi-resolution frame pack technique, CogVideoX is adept at producing coherent, long-duration, different shape videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method, greatly contributing to the generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weight of both 3D Causal VAE, Video caption model and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.
Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers:Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Maryland(马里兰大学);Nvidia(英伟达);Salesforce AI Research(Salesforce AI研究);Intuit AI Research(Intuit AI研究)
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Independent Researcher(独立研究员);University of Macau(澳门大学);The Chinese University of Hong Kong(香港中文大学)
Training-Free Text-Guided Color Editing with Multi-Modal Diffusion Transformer
The Hong Kong University of Science and Technology(香港科学与技术大学);International Digital Economy Academy(国际数字经济学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Tsinghua University(清华大学);Astribot;StepFun
ExFusion: Efficient Transformer Training via Multi-Experts Fusion
School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Huazhong University of Science and Technology(华中科技大学);Soochow University(苏州大学);Chinese University of Hong Kong(香港中文大学);School of Biomedical Engineering, Shanghai Jiao Tong University(上海交通大学生物医学工程学院)
ISALux: Illumination and Segmentation Aware Transformer Employing Mixture of Experts for Low Light Image Enhancement
Department of Computer Science University of Manchester(计算机科学系曼彻斯特大学);Department of Computer and Information Technology Politehnica University of Timisoara(计算机与信息科技系蒂森纳理工大学)
M$^3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design
University of Texas at Austin(德克萨斯大学奥斯汀分校);Georgia Institute of Technology(佐治亚理工学院);Texas A&M University(德克萨斯农工大学);Protagolabs Inc(Protagolabs公司);Microsoft Research(微软研究院)
LGEST: Dynamic Spatial-Spectral Expert Routing for Hyperspectral Image Classification
Information Hub, The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学信息中心(广州));Faculty of Geographical Science, Beijing Normal University(北京师范大学地理学部);School of Electronic and Communication Engineering, Guangzhou University(广州大学电子与通信工程学院)
MoCTEFuse: Illumination-Gated Mixture of Chiral Transformer Experts for Multi-Level Infrared and Visible Image Fusion
School of Optics and Photonics, Beijing Institute of Technology(光学与光电学院,北京理工大学);School of Computer Science and Technology, Beijing Institute of Technology(计算机科学与技术学院,北京理工大学)
MoECa: Aligning Feature Reuse with Expert Decomposition in Diffusion Transformers
School of Computer Science, Peking University(北京大学计算机科学学院);School of Software Engineering, University of Electronic Science and Technology of China(电子科技大学软件工程学院)
Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance
Fudan University(复旦大学);Tongyi Lab, Alibaba Group(阿里云实验室,阿里巴巴集团);Zhejiang University(浙江大学);The University of Hong Kong(香港大学);MMLab;Institute of Science and Technology for Brain-inspired Intelligence, MOE Frontiers Center for Brain Science, Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, and State Key Laboratory of Brain Function and Disorders, Fudan University(脑启发智能科学技术研究院、MOE前沿脑科学中心、计算神经科学与脑启发智能重点实验室、脑功能与疾病国家重点实验室,复旦大学)
SMGeo: Cross-View Object Geo-Localization with Grid-Level Mixture-of-Experts
School of Information Science and Technology(信息科学与技术学院);Interdisciplinary Research Center for Artificial Intelligence(人工智能交叉研究中心);Beijing University of Chemical Technology(北京化工大学)
MMOE: Modernizing Diffusion Transformers with Efficient Expert Design
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Institute of Artificial Intelligence of China Telecom (TeleAI)(中国电信人工智能研究院)
Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning
Northwestern Polytechnical University(西北工业大学);University of Georgia(佐治亚大学);University of Electronic Science and Technology of China(电子科技大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校);Athens Academy(雅典学院);Shanghai Jiao Tong University(上海交通大学);ShanghaiTech University(上海科技大学)
Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography
Concordia Institute for Information Systems Engineering (CIISE)(康科德信息系统工程研究所);Concordia University(康科德大学);Thomas Jefferson University Hospital(托马斯·杰斐逊大学医院)
Experts Weights Averaging: A New General Training Scheme for Vision Transformers
School of Information Science and Technology, Fudan University(复旦大学信息科学与技术学院);Shanghai AI Lab(上海人工智能实验室);Jiangxi University of Finance and Economics(江西财经大学)
Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking
School of Computer Science and Technology, Anhui University(安徽大学计算机科学与技术学院);Beijing Institute of Technology(北京理工大学);Peng Cheng Laboratory(鹏城实验室);National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(北京大学多媒体信息处理国家重点实验室);School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University(北京大学深圳研究生院电子与计算机工程学院)
NeuroMoE: A Transformer-Based Mixture-of-Experts Framework for Multi-Modal Neurological Disorder Classification
Department of Electrical and Computer Engineering, University of Houston(电气与计算机工程系,休斯顿大学);Department of Information Science Technology, University of Houston(信息科学与技术系,休斯顿大学);Movement Disorders & Neurodegenerative Diseases Division, University of Texas Health Science Center at Houston(运动障碍与神经退行性疾病部门,德克萨斯大学健康科学中心休斯顿分部);Department of Neurology at McGovern Medical School, University of Texas Health Science Center at Houston(神经病学系,麦戈文医学学校,德克萨斯大学健康科学中心休斯顿分部);Department of Psychology, The City College of New York(心理学系,纽约城市大学)