Engagement recognition in video datasets, unlike traditional image classification tasks, is particularly challenged by subjective labels and noise limiting model performance. To overcome the challenges of subjective and noisy engagement labels, we propose a framework leveraging Vision Large Language Models (VLMs) to refine annotations and guide the training process. Our framework uses a questionnaire to extract behavioral cues and split data into high- and low-reliability subsets. We also introduce a training strategy combining curriculum learning with soft label refinement, gradually incorporating ambiguous samples while adjusting supervision to reflect uncertainty. We demonstrate that classical computer vision models trained on refined high-reliability subsets and enhanced with our curriculum strategy show improvements, highlighting benefits of addressing label subjectivity with VLMs. This method surpasses prior state of the art across engagement benchmarks such as EngageNet (three of six feature settings, maximum improvement of +1.21%), and DREAMS / PAFE with F1 gains of +0.22 / +0.06.
Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);Shenzhen Polytechnic University(深圳职业技术大学);ZTE Corporation(中兴通讯股份有限公司);Peng Cheng Laboratory(鹏城实验室)
ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning
National University of Singapore(新加坡国立大学);Zhejiang University(浙江大学);University of Washington(华盛顿大学);Stanford University(斯坦福大学);absolute AI;The Chinese University of Hong Kong(香港中文大学)
Enhancing the Learning Experience: Using Vision-Language Models to Generate Questions for Educational Videos
TIB – Leibniz Information Centre for Science and Technology(蒂宾根-莱比锡信息科学与技术研究中心);Hochschule Hannover – Data(汉诺威高等学院-数据);H Institute for Applied Data Science(汉诺威应用数据科学研究所);L3S Research Center – Leibniz University Hannover(L3S研究中心-汉诺威莱比锡大学)
Neuroscience-Inspired Analyses of Visual Interestingness in Multimodal Transformers
Cognitive Computational Neuroscience Group, Patter Recognition Lab, University Erlangen-Nürnberg(认知计算神经科学组、模式识别实验室、埃尔兰根-纽伦堡大学);IDS Institut für Data Science, ZHAW School of Engineering, Winterthur, Switzerland(IDS数据科学研究所、ZHAW工程学院、温特图尔,瑞士);Mannheim Center for Neuromodulation and Neuroprosthetics, University Hospital Mannheim, Heidelberg University(曼海姆神经调制与神经假体中心、曼海姆大学医院、海德堡大学);BGU Ludwigshafen, Germany(BGU路易斯港,德国);Physics and Cognition Group, MCNN, University Hospital Mannheim, Heidelberg University(物理与认知组、MCNN、曼海姆大学医院、海德堡大学);NeuroAI and BCI Group, MCNN, University Hospital Mannheim, Heidelberg University(神经AI与BCI组、MCNN、曼海姆大学医院、海德堡大学)