arXiv 2507.186752025-07-30cs.CVcs.LG

推进基于视觉的人类动作识别：探索视觉语言CLIP模型在跨领域无关任务中的泛化能力

Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks

Central University of Haryana（哈里亚纳中央大学）；University of New South Wales（新南威尔士大学）；Østfold University College（东福尔大学学院）；Norwegian University of Science and Technology（挪威科技大学）

Utkarsh Shandilya, Marsha Mariya Kappan, Sanyam Jain, Vijeta Sharma

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究针对传统动作识别模型泛化能力不足的问题，在UCF-101数据集上评估CLIP模型在三类掩码策略下的性能，发现其存在误分类问题，进而提出引入类别特定噪声的改进方法，提升了准确率与置信度并降低偏差，为医疗场景动作识别泛化研究提供方向。

摘要：

人类动作识别在医疗健康与医学领域发挥着关键作用，可支撑患者行为监测、跌倒检测、手术机器人监督以及操作技能评估等应用。尽管CNN、RNN等传统模型已取得一定成效，但它们往往难以在多样复杂的动作场景中实现泛化。视觉语言模型的最新进展，尤其是基于Transformer的CLIP模型，为视频数据下的动作识别泛化提供了颇具潜力的能力。\n在本研究中，我们在UCF-101数据集上对CLIP进行评估，并系统分析其在三种掩码策略下的性能：（1）比例为10%、30%、50%的基于比例和基于形状的黑色掩码；（2）用于抑制偏差诱发元素的特征特定掩码；（3）仅保留类别相关区域的隔离掩码。结果显示，CLIP表现出行为不一致性且频繁出现误分类，尤其是在关键视觉线索被遮挡的情况下。\n为克服这些局限，我们提出引入通过自定义损失函数学习的类别特定噪声，以强化模型对类别定义特征的注意力。这一改进提升了分类准确率和模型置信度，同时降低了偏差。最后，我们讨论了此类模型应用于临床领域面临的挑战，并概述了提升其在跨领域无关医疗场景中泛化能力的未来研究方向。

英文摘要：

Human action recognition plays a critical role in healthcare and medicine, supporting applications such as patient behavior monitoring, fall detection, surgical robot supervision, and procedural skill assessment. While traditional models like CNNs and RNNs have achieved moderate success, they often struggle to generalize across diverse and complex actions. Recent advancements in vision-language models, especially the transformer-based CLIP model, offer promising capabilities for generalizing action recognition from video data. In this work, we evaluate CLIP on the UCF-101 dataset and systematically analyze its performance under three masking strategies: (1) percentage-based and shape-based black masking at 10%, 30%, and 50%, (2) feature-specific masking to suppress bias-inducing elements, and (3) isolation masking that retains only class-specific regions. Our results reveal that CLIP exhibits inconsistent behavior and frequent misclassifications, particularly when essential visual cues are obscured. To overcome these limitations, we propose incorporating class-specific noise, learned via a custom loss function, to reinforce attention to class-defining features. This enhancement improves classification accuracy and model confidence while reducing bias. We conclude with a discussion on the challenges of applying such models in clinical domains and outline directions for future work to improve generalizability across domain-independent healthcare scenarios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。