arXiv 2505.056812025-05-08cs.CV

微调视频-文本对比模型用于从未标注的原始视频中检索灵长类动物行为

Fine-Tuning Video-Text Contrastive Model for Primate Behavior Retrieval from Unlabeled Raw Videos

Institute of Mathematics and Statistics, University of São Paulo (IME-USP)（数学统计研究所，圣保罗大学）；Department of Experimental Psychology, Institute of Psychology, University of São Paulo (IP-USP)（心理学实验部门，心理学研究所，圣保罗大学）；Institute of Biosciences, University of São Paulo (IB-USP)（生物科学研究所，圣保罗大学）

Giulio Cesare Mastrocinque Santo, Patrícia Izar, Irene Delval, Victor de Napole Gregolin, Nina S. T. Hirata

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文针对未标注灵长类动物原始视频，提出智能体式数据处理流程与LoRA微调X-CLIP模型的方法，显著提升行为检索性能。

摘要：

非人灵长类动物在其自然栖息地中的视频记录是研究其野外行为的常见来源。我们针对卷尾猴这一特定领域微调预训练的视频-文本基础模型，目标是开发有用的计算模型，帮助研究人员从视频中检索有用的片段。我们专注于一个具有挑战性的问题：仅基于原始的、未标注的视频片段训练模型，并利用野外合作者有时提供的弱音频描述。我们利用多模态大语言模型（MLLMs）和视觉-语言模型（VLMs）的最新进展，来解决视频和音频内容中极其嘈杂的问题。具体来说，我们提出了一种双重方法：一个智能体式数据处理流程和一个微调过程。该数据处理流程自动从原始视频中提取干净且语义对齐的视频-文本对，随后通过低秩适应（LoRA）微调预训练的微软X-CLIP模型。在我们的领域数据上，16帧模型的Hits@5提升了167%，8帧模型提升了114%。此外，基于NDCG@K结果，我们的模型能够对大多数所考虑的行为进行良好排序，而测试的原始预训练模型则完全无法对它们进行排序。代码将在录用后公开。

英文摘要：

Video recordings of nonhuman primates in their natural habitat are a common source for studying their behavior in the wild. We fine-tune pre-trained video-text foundational models for the specific domain of capuchin monkeys, with the goal of developing useful computational models to help researchers to retrieve useful clips from videos. We focus on the challenging problem of training a model based solely on raw, unlabeled video footage, using weak audio descriptions sometimes provided by field collaborators. We leverage recent advances in Multimodal Large Language Models (MLLMs) and Vision-Language Models (VLMs) to address the extremely noisy nature of both video and audio content. Specifically, we propose a two-folded approach: an agentic data treatment pipeline and a fine-tuning process. The data processing pipeline automatically extracts clean and semantically aligned video-text pairs from the raw videos, which are subsequently used to fine-tune a pre-trained Microsoft's X-CLIP model through Low-Rank Adaptation (LoRA). We obtained an uplift in $Hits@5$ of $167\%$ for the 16 frames model and an uplift of $114\%$ for the 8 frame model on our domain data. Moreover, based on $NDCG@K$ results, our model is able to rank well most of the considered behaviors, while the tested raw pre-trained models are not able to rank them at all. The code will be made available upon acceptance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。