CG-MLLM：通过多模态大语言模型实现图像描述与3D内容生成
CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models
AI总结：
本文提出CG-MLLM，一种能实现3D描述和高分辨率3D生成的多模态大语言模型，通过混合Transformer架构分离不同建模需求，结合预训练视觉语言模型与专用3D VAE潜在空间，提升3D生成质量与感知能力。
摘要：
大型语言模型(LLMs)已革新了文本生成和多模态感知，但其在3D内容生成方面的能力仍待探索。现有方法往往只能生成低分辨率网格或粗略结构代理，无法原生捕捉细粒度几何结构。本文提出CG-MLLM，一种新型多模态大语言模型，能够在单一框架内实现3D描述和高分辨率3D生成。通过混合Transformer架构，CG-MLLM分离了不同的建模需求，其中Token-level Autoregressive (TokenAR) Transformer处理token级内容，Block-level Autoregressive (BlockAR) Transformer处理块级内容。通过整合预训练的视觉语言骨干网络与专用3D VAE潜在空间，CG-MLLM促进了标准token与空间块之间的长上下文交互。实验结果表明，CG-MLLM在生成高保真3D对象方面显著优于现有MLLMs，有效将高分辨率3D内容创作带入主流LLM范式。此外，我们进一步发现，学习生成3D内容能够反向增强模型的基于图像的3D理解能力。
英文摘要：
Large Language Models(LLMs) have revolutionized text generation and multimodal perception,but their capabilities in 3D content generation remain underexplored. Existing methods compromise by producing either low-resolution meshes or coarse structural proxies, failing to capture finegrained geometry natively. In this paper, we propose CG-MLLM, a novel Multi-modal Large Language Model (MLLM) capable of 3D captioning and high-resolution 3D generation in a single framework. Leveraging the Mixture-ofTransformer architecture, CG-MLLM decouples disparate modeling needs, where the Token-level Autoregressive (TokenAR) Transformer handles token-level content, and the Block-level Autoregressive (BlockAR) Transformer handles blocklevel content. By integrating a pre-trained visionlanguage backbone with a specialized 3D VAE latent space, CG-MLLM facilitates long-context interactions between standard tokens and spatial blocks within a single integrated architecture. Experimental results show that CG-MLLM significantly outperforms existing MLLMs in generating high-fidelity 3D objects, effectively bringing high-resolution 3D content creation into the mainstream LLM paradigm. Beyond generation, we further observe that learning to produce 3D content transfers back to perception, strengthening the model's image-based 3D understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
