arXiv 2602.080572026-02-08cs.CVcs.AI

弱到强：基于VLM的伪标签作为多模态视频隐藏情绪理解任务中的弱监督训练策略

Weak to Strong: VLM-Based Pseudo-Labeling as a Weakly Supervised Training Strategy in Multimodal Video-based Hidden Emotion Understanding Tasks

The University of New South Wales, Sydney, New South Wales, Australia（新南威尔士大学）；The University of Sydney, Sydney, New South Wales, Australia（悉尼大学）；School of Software and Microelectronics, Peking University, Zibo, Shandong, China（北京大学软件与微电子学院）；Shandong University of Technology, Beijing, China（山东理工大学）

Yufei Wang, Haixu Liu, Tianxiang Xu, Chuancheng Shi, Hongsheng Xing

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于VLM的伪标签弱监督方法，用于多模态视频隐藏情绪识别，提升准确率至0.69并建立新基准。

摘要：

为解决视频中'隐藏情绪'的自动识别问题，本文提出了一种多模态弱监督框架，并在iMiGUE网球采访数据集上取得了最先进的结果。首先，YOLO 11x每帧检测并裁剪人物肖像，DINOv2-Base从裁剪区域提取视觉特征。接着，通过整合Chain-of-Thought和Reflection提示（CoT + Reflection），Gemini 2.5 Pro自动生成伪标签和推理文本，作为下游模型的弱监督。随后，OpenPose生成137维的关键点序列，并补充了帧间偏移特征；通常的图神经网络主干被简化为MLP，以高效建模三个关键点流的时空关系。一个超长序列Transformer独立编码图像和关键点序列，其表示与BERT编码的采访 transcripts 连接。每种模态首先单独预训练，然后联合微调，伪标签样本合并到训练集中以进一步提升性能。实验表明，尽管存在严重的类别不平衡，所提出的方法将准确性从先前工作的不足0.6提升到超过0.69，建立了新的公开基准。该研究还验证了在该任务中，

英文摘要：

To tackle the automatic recognition of "concealed emotions" in videos, this paper proposes a multimodal weak-supervision framework and achieves state-of-the-art results on the iMiGUE tennis-interview dataset. First, YOLO 11x detects and crops human portraits frame-by-frame, and DINOv2-Base extracts visual features from the cropped regions. Next, by integrating Chain-of-Thought and Reflection prompting (CoT + Reflection), Gemini 2.5 Pro automatically generates pseudo-labels and reasoning texts that serve as weak supervision for downstream models. Subsequently, OpenPose produces 137-dimensional key-point sequences, augmented with inter-frame offset features; the usual graph neural network backbone is simplified to an MLP to efficiently model the spatiotemporal relationships of the three key-point streams. An ultra-long-sequence Transformer independently encodes both the image and key-point sequences, and their representations are concatenated with BERT-encoded interview transcripts. Each modality is first pre-trained in isolation, then fine-tuned jointly, with pseudo-labeled samples merged into the training set for further gains. Experiments demonstrate that, despite severe class imbalance, the proposed approach lifts accuracy from under 0.6 in prior work to over 0.69, establishing a new public benchmark. The study also validates that an "MLP-ified" key-point backbone can match - or even surpass - GCN-based counterparts in this task.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。