Multi-modal Auto-regressive Modeling via Visual Words
School of Computer Science, Wuhan University（武汉大学计算机学院）；Shanghai Jiao Tong University（上海交通大学）；School of Information Management, Wuhan University（武汉大学信息管理学院）
Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, Bo Du
Large Language Models (LLMs), benefiting from the auto-regressive modelling approach performed on massive unannotated texts corpora, demonstrates powerful perceptual and reasoning capabilities. However, as for extending auto-regressive modelling to multi-modal scenarios to build Large Multi-modal Models (LMMs), there lies a great difficulty that the image information is processed in the LMM as continuous visual embeddings, which cannot obtain discrete supervised labels for classification.In this paper, we successfully perform multi-modal auto-regressive modeling with a unified objective for the first time.Specifically, we propose the concept of visual tokens, which maps the visual features to probability distributions over LLM's vocabulary, providing supervision information for visual modelling.We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information.Experimental results and ablation studies on 5 VQA tasks and 4 benchmark toolkits validate the powerful performance of our proposed approach.
X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Models
Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；The Chinese University of Hong Kong（香港中文大学）；CPII under InnoHK（创新香港科技促进会）；MThreads AI
Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Sampling
Dept. Computing Science & Alberta Machine Intelligence Institute (Amii), University of Alberta（阿尔伯塔大学计算机科学系与阿尔伯塔机器智能研究所）；Canada CIFAR AI Chair（加拿大CIFAR人工智能讲席）
ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Shenzhen Loop Area Institute（深圳河套学院）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东省大数据分析与处理重点实验室）；Baidu Inc.（百度公司）；LightSpeed Studios, Tencent America（美国光速工作室，腾讯）；School of Computer Science and Artificial Intelligence, Lanzhou University of Technology（兰州理工大学计算机科学与人工智能学院）
High-Fidelity Functional Ultrasound Reconstruction via A Visual Auto-Regressive Framework
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Department of Computer and Information Science, University of Macau（计算机与信息科学系，澳门大学）
Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；CUHK（香港中文大学）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Infinigence AI（智源人工智能）
Quality-Aware Language-Conditioned Local Auto-Regressive Anomaly Synthesis and Detection
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（自动化研究所基础模型研究中心，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Objecteye Inc.（Objecteye公司）
SymDrive: Realistic and Controllable Driving Simulator via Symmetric Auto-regressive Online Restoration
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；State Key Lab of Intelligent Transportation System, Beihang University（北航智能交通系统国家重点实验室）
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and Scalability
Fudan University（复旦大学）；Fudan ISTBI—ZJNU Algorithm Centre for Brain-inspired Intelligence（复旦大学ISTBI—浙江师范大学类脑智能算法中心）；Tsinghua University（清华大学）；Google（谷歌公司）；Nanjing University（南京大学）；Tencent Youtu lab（腾讯优图实验室）；BAAI (Beijing Academy of Artificial Intelligence)（北京智源人工智能研究院）
XSpecMesh: Quality-Preserving Auto-Regressive Mesh Generation Acceleration via Multi-Head Speculative Decoding
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Shandong Inspur Database Technology Co.,Ltd（山东浪潮数据库技术有限公司）
GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photography
Zhejiang University（浙江大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Stanford University（斯坦福大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）
SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation
The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Peking University（北京大学）；xAI；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）
