Recent advancements in video generation have primarily leveraged diffusion models for short-duration content. However, these approaches often fall short in modeling complex narratives and maintaining character consistency over extended periods, which is essential for long-form video production like movies. We propose MovieDreamer, a novel hierarchical framework that integrates the strengths of autoregressive models with diffusion-based rendering to pioneer long-duration video generation with intricate plot progressions and high visual fidelity. Our approach utilizes autoregressive models for global narrative coherence, predicting sequences of visual tokens that are subsequently transformed into high-quality video frames through diffusion rendering. This method is akin to traditional movie production processes, where complex stories are factorized down into manageable scene capturing. Further, we employ a multimodal script that enriches scene descriptions with detailed character information and visual style, enhancing continuity and character identity across scenes. We present extensive experiments across various movie genres, demonstrating that our approach not only achieves superior visual and narrative quality but also effectively extends the duration of generated content significantly beyond current capabilities. Homepage: https://aim-uofa.github.io/MovieDreamer/.
Visual data and text data are composed of information at multiple granularities. A video can describe a complex scene that is composed of multiple clips or shots, where each depicts a semantically coherent event or action. Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story. In this paper, we investigate the modeling techniques for such hierarchical sequential data where there are correspondences across multiple modalities. Specifically, we introduce hierarchical sequence embedding (HSE), a generic model for embedding seque…
Microsoft Research（微软研究院）；University of Florida（佛罗里达大学）；JD AI Research（京东人工智能研究院）
We propose a hierarchically structured reinforcement learning approach to address the challenges of planning for generating coherent multi-sentence stories for the visual storytelling task. Within our framework, the task of generating a story given a sequence of images is divided across a two-level hierarchical decoder. The high-level decoder constructs a plan by generating a semantic concept (i.e., topic) for each image in sequence. The low-level decoder generates a sentence for each image using a semantic compositional network, which effectively grounds the sentence generation conditioned on…
HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Pengcheng Laboratory（鹏城实验室）；Peking University Shenzhen Graduate School（北京大学深圳研究生院）
HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Department of General Surgery, Beijing Friendship Hospital, Capital Medical University（首都医科大学附属北京友谊医院普通外科）；Chinese PLA General Hospital（中国人民解放军总医院）
Xidian University（西安电子科技大学）；McCombs School of Business, The University of Texas at Austin（德克萨斯大学奥斯汀分校麦库姆斯商学院）
Observing a set of images and their corresponding paragraph-captions, a challenging task is to learn how to produce a semantically coherent paragraph to describe the visual content of an image. Inspired by recent successes in integrating semantic topics into this task, this paper develops a plug-and-play hierarchical-topic-guided image paragraph generation framework, which couples a visual extractor with a deep topic model to guide the learning of a language model. To capture the correlations between the image and text at multiple levels of abstraction and learn the semantic topics from images…
Shandong University（山东大学）；Tencent AI Lab（腾讯人工智能实验室）
In this paper, we propose a novel model with a hierarchical photo-scene encoder and a reconstructor for the task of album storytelling. The photo-scene encoder contains two sub-encoders, namely the photo and scene encoders, which are stacked together and behave hierarchically to fully exploit the structure information of the photos within an album. Specifically, the photo encoder generates semantic representation for each photo while exploiting temporal relationships among them. The scene encoder, relying on the obtained photo representations, is responsible for detecting the scene changes and…
HiDiGen: Hierarchical Diffusion for B-Rep Generation with Explicit Topological Constraints
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；School of Intelligent Systems Engineering, Shenzhen Campus of Sun Yat-sen University, China（中山大学深圳校区智能工程学院）
H2G: Hierarchy-Aware Hyperbolic Grouping for 3D Scenes
Department of Computer Science and Engineering, Korea University（韩国大学计算机科学与工程系）；Intelligence and Interaction Research Center, Korea Institute of Science and Technology（韩国科学技术院智能与交互研究中心）
CoTZero: Annotation-Free Human-Like Vision Reasoning via Hierarchical Synthetic CoT
University of Electronic Science and Technology of China（电子科技大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong MMLab（香港中文大学 MMLab）；The College of Computer Science and Technology（计算机科学与技术学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）
Disentangled Hierarchical VAE for 3D Human-Human Interaction Generation
Department of CSSE, The University of Western Australia（西澳大学计算机科学与工程系）；Data61, CSIRO（澳大利亚联邦科学工业研究组织Data61部门）；Australian Institute for Machine Learning, The University of Adelaide（澳大利亚阿德莱德大学人工智能研究所）；NERC-RVC, Hunan University（湖南大学NERC-RVC部门）
Recently, medical report generation, which aims to automatically generate a long and coherent descriptive paragraph of a given medical image, has received growing research interests. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the serious data bias: the normal visual regions dominate the dataset over the abnormal visual regions, and 2) the very long sequence. To alleviate above two problems, we propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) …
