从舌尖上的遗忘检索查询中无监督建模可记忆性
Unsupervised Memorability Modeling from Tip-of-the-Tongue Retrieval Queries
AI总结：
本文提出首个大规模无监督数据集，通过舌尖上的遗忘检索查询建模视觉内容的可记忆性，并展示了其在回忆生成和ToT检索任务中的优越表现。
摘要：
视觉内容的可记忆性已经吸引了科学界数十年的关注，其应用范围广泛，从理解人类记忆的细微方面到增强内容设计。在推进该领域的一个重大挑战在于收集可记忆性标注的过程成本高昂。这限制了用于建模视觉内容可记忆性的数据集的多样性和可扩展性。大多数现有数据集仅限于收集视觉内容的总体可记忆性评分，而不是捕捉自然、开放式回忆描述中存在的细微可记忆性信号。在本工作中，我们引入了第一个大规模的无监督数据集，专门设计用于建模视觉可记忆性信号，包含超过82,000个视频，并附有描述性回忆数据。我们利用来自Reddit等在线平台的舌尖上的遗忘（ToT）检索查询。我们证明了我们的无监督数据集为两个可记忆性相关任务提供了丰富的信号：回忆生成和ToT检索。在我们的数据集上微调的大型视觉-语言模型在生成视觉内容的开放式可记忆性描述方面优于最先进的模型，如GPT-4o。我们还采用对比训练策略来创建第一个能够执行多模态ToT检索的模型。我们的数据集和模型为视觉内容可记忆性研究开辟了新的方向。
英文摘要：
Visual content memorability has intrigued the scientific community for decades, with applications ranging widely, from understanding nuanced aspects of human memory to enhancing content design. A significant challenge in progressing the field lies in the expensive process of collecting memorability annotations from humans. This limits the diversity and scalability of datasets for modeling visual content memorability. Most existing datasets are limited to collecting aggregate memorability scores for visual content, not capturing the nuanced memorability signals present in natural, open-ended recall descriptions. In this work, we introduce the first large-scale unsupervised dataset designed explicitly for modeling visual memorability signals, containing over 82,000 videos, accompanied by descriptive recall data. We leverage tip-of-the-tongue (ToT) retrieval queries from online platforms such as Reddit. We demonstrate that our unsupervised dataset provides rich signals for two memorability-related tasks: recall generation and ToT retrieval. Large vision-language models fine-tuned on our dataset outperform state-of-the-art models such as GPT-4o in generating open-ended memorability descriptions for visual content. We also employ a contrastive training strategy to create the first model capable of performing multimodal ToT retrieval. Our dataset and models present a novel direction, facilitating progress in visual content memorability research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
