告诉我你看到了什么：一种基于自然语言描述的零样本动作识别方法
Tell me what you see: A zero-shot action recognition method based on natural language descriptions
AI总结：
本文提出一种基于自然语言描述的零样本动作识别方法，通过视频描述生成和搜索引擎文档提取构建视频与标签的句子表示，并利用预训练BERT嵌入器构建共享语义空间进行最近邻分类，在UCF101和HMDB51数据集上取得优异结果。
摘要：
本文提出了一种新的零样本动作识别（Zero-Shot Action Recognition）方法。近期的研究探索了物体的检测与分类，以从视频中获取语义信息，并取得了显著的性能。受此启发，我们提出使用视频描述生成方法来提取关于物体、场景、人类及其关系的语义信息。据我们所知，这是首个使用描述性句子来同时表示视频和标签的工作。更具体地说，我们使用通过视频描述生成方法生成的句子来表示视频，并使用从互联网搜索引擎获取的文档中提取的句子来表示类别。利用这些表示，我们采用在多个文本数据集上的释义任务中预训练的基于BERT的嵌入器，构建了一个共享语义空间。由于视觉和语义信息都是句子，将它们投影到该空间的过程非常直接，从而能够使用最近邻规则进行分类。我们证明，用句子表示视频和标签缓解了领域适应问题。此外，我们表明词向量不适合构建我们描述的语义嵌入空间。我们的方法在TruZe协议下的UCF101数据集上以3.3个百分点的准确率优势超越了现有最先进的性能，并在常规协议（0/50% - 训练/测试划分）下的UCF101和HMDB51数据集上取得了具有竞争力的结果。我们的代码可在 https://github.com/valterlej/zsarcap 获取。
英文摘要：
This paper presents a novel approach to Zero-Shot Action Recognition. Recent works have explored the detection and classification of objects to obtain semantic information from videos with remarkable performance. Inspired by them, we propose using video captioning methods to extract semantic information about objects, scenes, humans, and their relationships. To the best of our knowledge, this is the first work to represent both videos and labels with descriptive sentences. More specifically, we represent videos using sentences generated via video captioning methods and classes using sentences extracted from documents acquired through search engines on the Internet. Using these representations, we build a shared semantic space employing BERT-based embedders pre-trained in the paraphrasing task on multiple text datasets. The projection of both visual and semantic information onto this space is straightforward, as they are sentences, enabling classification using the nearest neighbor rule. We demonstrate that representing videos and labels with sentences alleviates the domain adaptation problem. Additionally, we show that word vectors are unsuitable for building the semantic embedding space of our descriptions. Our method outperforms the state-of-the-art performance on the UCF101 dataset by 3.3 p.p. in accuracy under the TruZe protocol and achieves competitive results on both the UCF101 and HMDB51 datasets under the conventional protocol (0/50\% - training/testing split). Our code is available at https://github.com/valterlej/zsarcap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
