OVC-Net: Object-Oriented Video Captioning with Temporal Graph and Detail Enhancement
School of Artificial Intelligence Beijing Univerisity of Posts\ Telecommunications Beijing, China Email；Department of Electrical；School of Artificial Intelligence Beijing Univerisity of Posts
Fangyi Zhu, Jenq-Neng Hwang, Zhanyu Ma, Guang Chen, Jun Guo
Traditional video captioning requests a holistic description of the video, yet the detailed descriptions of the specific objects may not be available. Without associating the moving trajectories, these image-based data-driven methods cannot understand the activities from the spatio-temporal transitions in the inter-object visual features. Besides, adopting ambiguous clip-sentence pairs in training, it goes against learning the multi-modal functional mappings owing to the one-to-many nature. In this paper, we propose a novel task to understand the videos in object-level, named object-oriented video captioning. We introduce the video-based object-oriented video captioning network (OVC)-Net via temporal graph and detail enhancement to effectively analyze the activities along time and stably capture the vision-language connections under small-sample condition. The temporal graph provides useful supplement over previous image-based approaches, allowing to reason the activities from the temporal evolution of visual features and the dynamic movement of spatial locations. The detail enhancement helps to capture the discriminative features among different objects, with which the subsequent captioning module can yield more informative and precise descriptions. Thereafter, we construct a new dataset, providing consistent object-sentence pairs, to facilitate effective cross-modal learning. To demonstrate the effectiveness, we conduct experiments on the new dataset and compare it with the state-of-the-art video captioning methods. From the experimental results, the OVC-Net exhibits the ability of precisely describing the concurrent objects, and achieves the state-of-the-art performance.
Video captioning combines video understanding and language generation. Different from image captioning that describes a static image with details of almost every object, video captioning usually considers a sequence of frames and biases towards focused objects, e.g., the objects that stay in focus regardless of the changing background. Therefore, detecting and properly accommodating focused objects is critical in video captioning. To enforce the description of focused objects and achieve controllable video captioning, we propose an Object-Oriented Non-Autoregressive approach (O2NA), which perf…
IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning
University of Science and Technology of China（中国科学技术大学）；Hefei Institutes of Physical Science,Chinese Academy of Sciences（合肥物理研究所，中国科学院）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究所，合肥综合性国家科学中心）；State Key Lab. for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；University of Chicago（芝加哥大学）；National University of Defense Technology（国防科技大学）；Harbin Institute of Technology（哈尔滨工业大学）
University of Electronic Science and Technology of China（电子科技大学）
In the dataset of image captioning, each image is aligned with several descriptions. Despite the fact that the quality of these descriptions varies, existing captioning models treat them equally in the training process. In this paper, we propose a new control signal of sentence quality, which is taken as an additional input to the captioning model. By integrating the control signal information, captioning models are aware of the quality level of the target sentences and handle them differently. Moreover, we propose a novel reinforcement training method specially designed for the control signal…
For video captioning, "pre-training and fine-tuning" has become a de facto paradigm, where ImageNet Pre-training (INP) is usually used to encode the video content, then a task-oriented network is fine-tuned from scratch to cope with caption generation. This paper first investigates the impact of the recently proposed CLIP (Contrastive Language-Image Pre-training) on video captioning. Through the empirical study on INP vs. CLIP, we identify the potential deficiencies of INP and explore the key factors for accurate description generation. The results show that the INP-based model is tricky to ca…
Renmin University of China（中国人民大学）；Visionary Intelligence Ltd.（睿思智能有限公司）
Cross-lingual image captioning, with its ability to caption an unlabeled image in a target language other than English, is an emerging topic in the multimedia field. In order to save the precious human resource from re-writing reference sentences per target language, in this paper we make a brave attempt towards annotation-free evaluation of cross-lingual image captioning. Depending on whether we assume the availability of English references, two scenarios are investigated. For the first scenario with the references available, we propose two metrics, i.e., WMDRel and CLinRel. WMDRel measures t…
In e-commerce, a growing number of user-generated videos are used for product promotion. How to generate video descriptions that narrate the user-preferred product characteristics depicted in the video is vital for successful promoting. Traditional video captioning methods, which focus on routinely describing what exists and happens in a video, are not amenable for product-oriented video captioning. To address this problem, we propose a product-oriented video captioner framework, abbreviated as Poet. Poet firstly represents the videos as product-oriented spatial-temporal graphs. Then, based on…
ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting
School of Information（信息学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Department of Statistics and Data Science（统计与数据科学系）；Yale University（耶鲁大学）；School of Computing and Augmented Intelligence（计算与增强智能学院）
ORCA: Object Recognition and Comprehension for Archiving Marine Species
Hong Kong University of Science and Technology（香港科技大学）；University of Electronic Science and Technology of China（电子科学与技术大学）；Indo Ocean Foundation Project（印度洋基金会项目）
HIMEC: Directional Change Representation and Fixed-Interface Decoding for Remote Sensing Image Change Captioning
University of Electronic Science and Technology of China（电子科技大学）；University of Bonn（波恩大学）；Bonn-Aachen International Center for Information Technology (B-IT)（波恩-亚琛信息技术国际中心（B-IT））
Shouldn't language and vision features be treated equally in vision-language (VL) tasks? Many VL approaches treat the language component as an afterthought, using simple language models that are either built upon fixed word embeddings trained on text-only data or are learned from scratch. We believe that language features deserve more attention, and conduct experiments which compare different word embeddings, language models, and embedding augmentation steps on five common VL tasks: image-sentence retrieval, image captioning, visual question answering, phrase grounding, and text-to-clip retrie…
Tianjin University（天津大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Image-to-text tasks, such as open-ended image captioning and controllable image description, have received extensive attention for decades. Here, we further advance this line of work by presenting Visual Spatial Description (VSD), a new perspective for image-to-text toward spatial semantics. Given an image and two objects inside it, VSD aims to produce one description focusing on the spatial perspective between the two objects. Accordingly, we manually annotate a dataset to facilitate the investigation of the newly-introduced task and build several benchmark encoder-decoder models by using VL-…
Captioning Daily Activity Images in Early Childhood Education: Benchmark and Algorithm
School of Robotics, Beijing Union University（北京联合大学机器人学院）；College of Computer and Cyber Security, Hebei Normal University（河北师范大学计算机与网络安全学院）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）；National Laboratory of Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；PeopleAI, Inc.（人民人工智能公司）；People Youhe Education Technology Co., Ltd.（人民优和教育科技有限公司）
Renmin University of China（中国人民大学）；Carnegie Mellon University（卡内基梅隆大学）
Generating video descriptions in natural language (a.k.a. video captioning) is a more challenging task than image captioning as the videos are intrinsically more complicated than images in two aspects. First, videos cover a broader range of topics, such as news, music, sports and so on. Second, multiple topics could coexist in the same video. In this paper, we propose a novel caption model, topic-guided model (TGM), to generate topic-oriented descriptions for videos in the wild via exploiting topic information. In addition to predefined topics, i.e., category tags crawled from the web, we also…
Beyond generating long and topic-coherent paragraphs in traditional captioning tasks, the medical image report composition task poses more task-oriented challenges by requiring both the highly-accurate medical term diagnosis and multiple heterogeneous forms of information including impression and findings. Current methods often generate the most common sentences due to dataset bias for individual case, regardless of whether the sentences properly capture key entities and relationships. Such limitations severely hinder their applicability and generalization capability in medical report composit…
