局部视觉-语言模型能否在视觉转换器上提高活动识别？——新生儿复苏案例研究
Can Local Vision-Language Models improve Activity Recognition over Vision Transformers? -- Case Study on Newborn Resuscitation
AI总结：
本研究探讨了局部视觉-语言模型在新生儿复苏视频活动识别中的应用，通过LoRA微调提升性能，达到0.91的F1分数，优于TimeSFormer基线。
摘要：
新生儿复苏过程的准确记录对于质量改进和遵守临床指南至关重要，但在实践中仍被低估。先前使用3D-CNN和视觉转换器（ViT）的工作在从新生儿复苏视频中检测关键活动方面显示出有希望的结果，但也突显了识别此类细粒度活动的挑战。本工作探讨了生成式人工智能（GenAI）方法在提高此类视频活动识别方面的潜力。具体来说，我们探索了局部视觉-语言模型（VLMs）与大型语言模型（LLMs）的结合使用，并将其与监督的TimeSFormer基线进行比较。使用包含13.26小时新生儿复苏视频的模拟数据集，我们评估了几种零样本VLM基于策略和带有分类头的微调VLMs，包括低秩适应（LoRA）。我们的结果表明，小型（局部）VLMs在处理时会出现幻觉，但当使用LoRA进行微调时，结果达到F1得分为0.91，超过了TimeSformer的结果0.70。
英文摘要：
Accurate documentation of newborn resuscitation is essential for quality improvement and adherence to clinical guidelines, yet remains underutilized in practice. Previous work using 3D-CNNs and Vision Transformers (ViT) has shown promising results in detecting key activities from newborn resuscitation videos, but also highlighted the challenges in recognizing such fine-grained activities. This work investigates the potential of generative AI (GenAI) methods to improve activity recognition from such videos. Specifically, we explore the use of local vision-language models (VLMs), combined with large language models (LLMs), and compare them to a supervised TimeSFormer baseline. Using a simulated dataset comprising 13.26 hours of newborn resuscitation videos, we evaluate several zero-shot VLM-based strategies and fine-tuned VLMs with classification heads, including Low-Rank Adaptation (LoRA). Our results suggest that small (local) VLMs struggle with hallucinations, but when fine-tuned with LoRA, the results reach F1 score at 0.91, surpassing the TimeSformer results of 0.70.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
