探索对比语言-图像预训练在人体姿态分类中的应用：从瑜伽姿势分析获得的见解
Exploring the Use of Contrastive Language-Image Pre-Training for Human Posture Classification: Insights from Yoga Pose Analysis
AI总结：
本研究利用CLIP模型在瑜伽姿势分类中取得高准确率，展示其在人体姿态识别中的潜力，并验证其在自动化系统中的应用可行性。
摘要：
准确的人体姿态图像和视频分类对于自动化应用至关重要，涵盖工作安全、物理康复、体育训练或日常辅助生活等多个领域。最近，多模态学习方法，如对比语言-图像预训练（CLIP），在联合理解图像和文本方面取得了显著进展。本研究旨在评估CLIP在人体姿态分类中的有效性，重点探讨其在瑜伽中的应用。尽管零样本方法最初存在局限性，但将转移学习应用于15,301张图像（真实和合成）并包含82个类别的数据集后，已显示出有希望的结果。本文描述了完整的微调过程，包括图像描述语法的选择、模型和超参数的调整。经过微调的CLIP模型在3,826张图像上实现了超过85%的准确率，超过了在同一数据集上先前工作的当前最佳水平，大约高出6%，其训练时间比微调YOLOv8模型所需的时间低3.5倍。对于更应用导向的场景，使用每个姿势6个样本的数据集，包含1,301和401张训练图像，微调后的模型分别达到98.8%和99.1%的准确率。此外，我们的实验表明，仅使用每个姿态20张图像进行训练，即可在六类数据集中获得约90%的准确率。本研究证明了这种多模态技术可以有效用于瑜伽姿势分类，并可能用于一般的人体姿态分类。此外，CLIP的推理时间（约7毫秒）表明该模型可以集成到自动化系统中用于姿态评估，例如开发实时个人瑜伽助手用于表现评估。
英文摘要：
Accurate human posture classification in images and videos is crucial for automated applications across various fields, including work safety, physical rehabilitation, sports training, or daily assisted living. Recently, multimodal learning methods, such as Contrastive Language-Image Pretraining (CLIP), have advanced significantly in jointly understanding images and text. This study aims to assess the effectiveness of CLIP in classifying human postures, focusing on its application in yoga. Despite the initial limitations of the zero-shot approach, applying transfer learning on 15,301 images (real and synthetic) with 82 classes has shown promising results. The article describes the full procedure for fine-tuning, including the choice for image description syntax, models and hyperparameters adjustment. The fine-tuned CLIP model, tested on 3826 images, achieves an accuracy of over 85%, surpassing the current state-of-the-art of previous works on the same dataset by approximately 6%, its training time being 3.5 times lower than what is needed to fine-tune a YOLOv8-based model. For more application-oriented scenarios, with smaller datasets of six postures each, containing 1301 and 401 training images, the fine-tuned models attain an accuracy of 98.8% and 99.1%, respectively. Furthermore, our experiments indicate that training with as few as 20 images per pose can yield around 90% accuracy in a six-class dataset. This study demonstrates that this multimodal technique can be effectively used for yoga pose classification, and possibly for human posture classification, in general. Additionally, CLIP inference time (around 7 ms) supports that the model can be integrated into automated systems for posture evaluation, e.g., for developing a real-time personal yoga assistant for performance assessment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
