arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.14749cs.CV
收藏

视觉大语言模型是参与度分析中的优秀噪声处理工具

Vision Large Language Models Are Good Noise Handlers in Engagement Analysis

Alexander Vedernikov, Puneet Kumar, Haoyu Chen, Tapio Seppänen, Xiaobai Li

AI总结:

针对视频参与度识别中主观标签和噪声的挑战,提出利用视觉大语言模型(VLMs)优化标注并结合课程学习与软标签优化的训练框架,提升经典模型性能并超越现有技术。

摘要:

视频数据集中的参与度识别与传统图像分类任务不同,其尤其受主观标签和噪声的挑战,限制了模型性能。为克服主观且带噪声的参与度标签带来的挑战,我们提出一种利用视觉大语言模型(VLMs)优化标注并指导训练过程的框架。该框架通过问卷提取行为线索,将数据分为高可靠性和低可靠性子集;还引入结合课程学习与软标签优化的训练策略,逐步纳入模糊样本,同时调整监督以反映不确定性。我们证明,在优化后的高可靠性子集上训练并结合该课程策略的经典计算机视觉模型性能提升,凸显了用VLMs解决标签主观性的优势;该方法在EngageNet(6个特征设置中的3个,最大提升+1.21%)、DREAMS/PAFE(F1值分别提升+0.22/+0.06)等参与度基准上超越现有技术。

英文摘要:

Engagement recognition in video datasets, unlike traditional image classification tasks, is particularly challenged by subjective labels and noise limiting model performance. To overcome the challenges of subjective and noisy engagement labels, we propose a framework leveraging Vision Large Language Models (VLMs) to refine annotations and guide the training process. Our framework uses a questionnaire to extract behavioral cues and split data into high- and low-reliability subsets. We also introduce a training strategy combining curriculum learning with soft label refinement, gradually incorporating ambiguous samples while adjusting supervision to reflect uncertainty. We demonstrate that classical computer vision models trained on refined high-reliability subsets and enhanced with our curriculum strategy show improvements, highlighting benefits of addressing label subjectivity with VLMs. This method surpasses prior state of the art across engagement benchmarks such as EngageNet (three of six feature settings, maximum improvement of +1.21%), and DREAMS / PAFE with F1 gains of +0.22 / +0.06.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑