User-given tags or labels are valuable resources for semantic understanding of visual media such as images and videos. Recently, a new type of labeling mechanism known as hash-tags have become increasingly popular on social media sites. In this paper, we study the problem of generating relevant and useful hash-tags for short video clips. Traditional data-driven approaches for tag enrichment and recommendation use direct visual similarity for label transfer and propagation. We attempt to learn a direct low-cost mapping from video to hash-tags using a two step training process. We first employ a natural language processing (NLP) technique, skip-gram models with neural network training to learn a low-dimensional vector representation of hash-tags (Tag2Vec) using a corpus of 10 million hash-tags. We then train an embedding function to map video features to the low-dimensional Tag2vec space. We learn this embedding for 29 categories of short video clips with hash-tags. A query video without any tag-information can then be directly mapped to the vector space of tags using the learned embedding and relevant tags can be found by performing a simple nearest-neighbor retrieval in the Tag2Vec space. We validate the relevance of the tags suggested by our system qualitatively and quantitatively with a user study.
Dual-level Semantic Transfer Deep Hashing for Efficient Social Image Retrieval
Shandong Normal University(山东师范大学);Shandong Computer Science Center (National Supercomputer Center in Jinan)(山东省计算中心(国家超级计算济南中心));Qilu University of Technology (Shandong Academy of Sciences)(齐鲁工业大学(山东省科学院));University of Electronic Science and Technology of China(电子科技大学);Harbin Institute of Technology(哈尔滨工业大学)
AutoSSVH: Exploring Automated Frame Sampling for Efficient Self-Supervised Video Hashing
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Research Center of Artificial Intelligence, Peng Cheng Laboratory(鹏城实验室人工智能研究中心)
STGV: Spatio-Temporal Hash Encoding for Gaussian-based Video Representation
College of Electronics and Information Engineering(电子与信息工程学院);College of Applied Technology(应用技术学院);School of Artificial Intelligence(人工智能学院);Research Center of Networks and Communications(网络与通信研究中心);Pengcheng Laboratory(鹏城实验室)
Efficient Self-Supervised Video Hashing with Selective State Spaces
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);Meituan(美团)
Discovering Visual Concept Structure with Sparse and Incomplete Tags
Queen Mary University of London(伦敦玛丽女王大学)
自动发现标记的视觉数据(例如网络视频和图像)的语义结构对于视觉数据分析和解释非常重要,使机器智能能够有效地处理快速增长的多媒体数据量。然而,这是不平凡的,因为需要共同学习异构视觉和标签数据之间的潜在相关性。由于标签本身稀疏和不完整,这项任务更具挑战性。在这项工作中,我们开发了一种新的层次多标签随机森林模型的基础上,能够相关的结构化视觉和标签信息,以便更准确地解释视觉语义,例如,披露有意义的视觉组与类似的高层次的概念,并恢复个别视觉数据样本丢失的标签的固有视觉数据概念结构建模的方法。具体来说,我们的模型利用层次结构的标签不同的语义抽象性和多个标签的统计相关性,除了建模视觉和标签的相互作用。因此,我们的模型能够发现文本标签和视觉特征之间更准确的语义相关性,并最终提供有利的视觉语义解释,即使是高度稀疏和不完整的标签。我们展示了我们提出的方法在两个基本应用程序中的优势,视觉数据聚类和缺失标签完成,基准视频(即TRECVID MED 2011)和图像(即NUS-WIDE)数据集。