首页/论文/1903.01489

arXiv 1903.014892019-03-04cs.CV

M-VAD Names: a Dataset for Video Captioning with Naming

University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）

Stefano Pini, Marcella Cornia, Federico Bolelli, Lorenzo Baraldi, Rita Cucchiara

备注：: Source Code: https://github.com/aimagelab/mvad-names-dataset - Video Demo: https://youtu.be/dOvtAXbOOH4
发表信息：: Multimedia Tools and Applications (2018)
DOI：: 10.1007/s11042-018-7040-z

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

Current movie captioning architectures are not capable of mentioning characters with their proper name, replacing them with a generic "someone" tag. The lack of movie description datasets with characters' visual annotations surely plays a relevant role in this shortage. Recently, we proposed to extend the M-VAD dataset by introducing such information. In this paper, we present an improved version of the dataset, namely M-VAD Names, and its semi-automatic annotation procedure. The resulting dataset contains 63k visual tracks and 34k textual mentions, all associated with character identities. To showcase the features of the dataset and quantify the complexity of the naming task, we investigate multimodal architectures to replace the "someone" tags with proper character names in existing video captions. The evaluation is further extended by testing this application on videos outside of the M-VAD Names dataset.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Video Captioning with Transferred Semantic Attributes

University of Science and Technology of China（中国科学技术大学）；Microsoft Research（微软研究院）

Automatically generating natural language descriptions of videos plays a fundamental challenge for computer vision community. Most recent progress in this problem has been achieved through employing 2-D and/or 3-D Convolutional Neural Networks (CNN) to encode video content and Recurrent Neural Networks (RNN) to decode a sentence. In this paper, we present Long Short-Term Memory with Transferred Semantic Attributes (LSTM-TSA)---a novel deep architecture that incorporates the transferred semantic attributes learnt from images and videos into the CNN plus RNN framework, by training them in an end…

2016-11-23cs.CV

COPRA：基于强化学习的条件参数适应用于视频异常检测

COPRA: Conditional Parameter Adaptation with Reinforcement Learning for Video Anomaly Detection

Auburn University（奥本大学）；Tencent Hunyuan（腾讯文元）

COPRA通过生成输入特定的参数更新，动态适应冻结的VLM，提升视频异常检测的适应性和泛化能力，同时拓展到多选视频问答和密集标注等任务。

2026-05-14cs.CV

PaLM2-VAdapter：渐进对齐的语言模型构成强大的视觉-语言适配器

PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter

Johns Hopkins University（约翰斯·霍普金斯大学）；Google Research（谷歌研究院）

提出PaLM2-VAdapter，采用渐进对齐的语言模型作为视觉-语言适配器，有效连接冻结视觉编码器与LLMs，在VQA和图像描述任务上以更少参数实现最先进的视觉理解与多模态推理能力。

2024-06-01cs.CV

面向新闻图像描述的视觉感知上下文建模

Visually-Aware Context Modeling for News Image Captioning

KU Leuven（KU Leuven（荷语鲁汶大学））

本文提出一种面向新闻图像描述的视觉感知上下文建模框架，包含人脸命名模块、基于CLIP的检索策略及CoLaM方法，在GoodNews和NYTimes800k数据集上显著超越现有方法。

2024-03-21cs.CV

通过细粒度情感-原因对提取实现精确的情感归因视频字幕生成

Towards Accurate Emotion-Attributed Video Captioning via Fine-grained Emotion-Cause Pair Extraction

University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Harbin Institute of Technology (Weihai)（哈尔滨工业大学（威海））

提出细粒度情感-原因对提取框架，通过概念感知视觉语义分解和视觉引导情感可解释学习，提升情感视频字幕的准确性和丰富性。

2026-06-07cs.CV

Visual-aware Attention Dual-stream Decoder for Video Captioning

Wuhan University of Technology（武汉理工大学）；ZhongQianLiYuan Engineering Consulting Co., Ltd（中前立源工程咨询有限公司）

Video captioning is a challenging task that captures different visual parts and describes them in sentences, for it requires visual and linguistic coherence. The attention mechanism in the current video captioning method learns to assign weight to each frame, promoting the decoder dynamically. This may not explicitly model the correlation and the temporal coherence of the visual features extracted in the sequence frames.To generate semantically coherent sentences, we propose a new Visual-aware Attention (VA) model, which concatenates dynamic changes of temporal sequence frames with the words a…

2021-10-16cs.AIcs.CV

更多相关论文收起更多论文

利用大型语言模型实现免训练的视频异常检测

Harnessing Large Language Models for Training-free Video Anomaly Detection

University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）

本文提出免训练的LAVAD方法，利用预训练大语言模型和视觉-语言模型，通过帧级字幕生成、提示机制与跨模态相似性细化实现视频异常检测，在UCF-Crime和XD-Violence上超越无监督与单类方法。

2024-04-01cs.CV

JoVALE：利用视听和语言上下文检测视频中的人类动作

JoVALE: Detecting Human Actions in Video Using Audiovisual and Language Contexts

Hanyang University（汉阳大学）；Seoul National University（首尔大学）

提出多模态VAD架构JoVALE，通过以动作者为中心的融合网络整合视听特征与图像描述模型生成的场景上下文，在AVA等基准上实现最先进性能。

2025-02-03cs.CV

Adaptive Feature Abstraction for Translating Video to Text

Duke University（杜克大学）；NEC Laboratories America（NEC美国实验室）

Previous models for video captioning often use the output from a specific layer of a Convolutional Neural Network (CNN) as video features. However, the variable context-dependent semantics in the video may make it more appropriate to adaptively select features from the multiple CNN layers. We propose a new approach for generating adaptive spatiotemporal representations of videos for the captioning task. A novel attention mechanism is developed, that adaptively and sequentially focuses on different layers of CNN features (levels of feature "abstraction"), as well as local spatiotemporal regions…

2017-11-17cs.CLcs.CV

Oracle performance for visual captioning

Université de Montréal（蒙特利尔大学）；IBM T.J. Watson Research（IBM T.J. Watson 研究院）；New York University（纽约大学）

The task of associating images and videos with a natural language description has attracted a great amount of attention recently. Rapid progress has been made in terms of both developing novel algorithms and releasing new datasets. Indeed, the state-of-the-art results on some of the standard datasets have been pushed into the regime where it has become more and more difficult to make significant improvements. Instead of proposing new models, this work investigates the possibility of empirically establishing performance upper bounds on various visual captioning datasets without extra data label…

2016-09-14cs.CLcs.CVstat.ML

AutoAD: Movie Description in Context

University of Oxford（牛津大学）；École des Ponts（巴黎桥路学院）；Univ Gustave Eiffel（古斯塔夫·埃菲尔大学）；CNRS（法国国家科学研究中心）；Shanghai Jiao Tong University（上海交通大学）

The objective of this paper is an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. Generating high-quality movie AD is challenging due to the dependency of the descriptions on context, and the limited amount of training data available. In this work, we leverage the power of pretrained foundation models, such as GPT and CLIP, and only train a mapping network that bridges the two models for visually-conditioned text generation. In order to obtain high-quality AD, we make the following four contributions: (i) we incorporate context from the movie clip, AD fr…

2023-03-29cs.CV

Title Generation for User Generated Videos

National Tsing Hua University（国立清华大学）；Stanford University（斯坦福大学）

A great video title describes the most salient event compactly and captures the viewer's attention. In contrast, video captioning tends to generate sentences that describe the video as a whole. Although generating a video title automatically is a very useful task, it is much less addressed than video captioning. We address video title generation for the first time by proposing two methods that extend state-of-the-art video captioners to this new task. First, we make video captioners highlight sensitive by priming them with a highlight detector. Our framework allows for jointly training a model…

2016-09-08cs.AIcs.CVcs.MM