Queries Are Not Alone: Clustering Text Embeddings for Video Search
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);Peking University(北京大学);City University of Hong Kong(香港城市大学)
The rapid proliferation of video content across various platforms has highlighted the urgent need for advanced video retrieval systems. Traditional methods, which primarily depend on directly matching textual queries with video metadata, often fail to bridge the semantic gap between text descriptions and the multifaceted nature of video content. This paper introduces a novel framework, the Video-Text Cluster (VTC), which enhances video retrieval by clustering text queries to capture a broader semantic scope. We propose a unique clustering mechanism that groups related queries, enabling our system to consider multiple interpretations and nuances of each query. This clustering is further refined by our innovative Sweeper module, which identifies and mitigates noise within these clusters. Additionally, we introduce the Video-Text Cluster-Attention (VTC-Att) mechanism, which dynamically adjusts focus within the clusters based on the video content, ensuring that the retrieval process emphasizes the most relevant textual features. Further experiments have demonstrated that our proposed model surpasses existing state-of-the-art models on five public datasets.
Efficient Media Retrieval from Non-Cooperative Queries
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);eBay Research Labs(eBay研究实验室)
文本在人工世界中无处不在,当涉及到书名和作者姓名时,文本很容易获得。使用来自Stanford Mobile Visual Search数据集的书籍封面集的图像以及来自openlibrary.org的其他书籍封面和元数据,我们构建了一个大规模的书籍封面检索数据集,每个数据集包含10万个干扰项封面以及标题和作者字符串。由于我们的查询图像是不好的条件,干净的文本提取,我们提出了一种方法,用于提取匹配的噪声和错误的OCR读数,并匹配它对干净的作者和书名字符串在一个标准的文档查找问题设置。最后,我们演示了如何使用这种文本匹配作为一个功能,结合流行的检索功能,如VLAD使用一个简单的学习设置,以实现检索精度的显着改善,无论是VLAD或单独的文本。