面向肢体语言识别与生成的深度多模态学习综述
A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation
AI总结：
本文综述深度多模态学习在SL、CS、CoS和TH四类肢体语言识别与生成中的应用，梳理数据集、SOTA方法、挑战与未来方向，并首次建立四类任务间的联系。
摘要：
肢体语言（BL）是指通过身体动作、手势、面部表情和姿势所表达的非语言交流。它是一种在不使用口头或书面语言的情况下传递信息、情绪、态度和意图的交流形式。它在人际互动中发挥着关键作用，能够补充甚至压倒语言交流。深度多模态学习技术在理解和分析BL的这些多样方面已展现出良好前景。本综述重点介绍它们在BL生成与识别中的应用。文中考虑了若干常见的BL，即手语（SL）、提示语（CS）、协同语音（CoS）和会说话头像（TH），并且我们首次开展分析并建立了这四种BL之间的联系。它们的生成和识别通常涉及多模态方法。文中完善地收集和整理了用于BL研究的基准数据集，并对这些数据集上的SOTA方法进行了评估。本综述强调了若干挑战，例如标注数据有限、多模态学习，以及需要领域自适应以使模型泛化到未见过的说话者或语言。文中提出了未来研究方向，包括探索自监督学习技术、整合来自其他模态的上下文信息，以及利用大规模预训练多模态模型。总之，本综述论文首次全面阐述了用于各类BL生成与识别的深度多模态学习。通过分析进展、挑战和未来方向，它为研究人员和从业者推进该领域提供了宝贵资源。此外，我们维护了一个持续更新的、关于面向BL识别与生成的深度多模态学习的论文列表：https://github.com/wentaoL86/awesome-body-language。
英文摘要：
Body language (BL) refers to the non-verbal communication expressed through physical movements, gestures, facial expressions, and postures. It is a form of communication that conveys information, emotions, attitudes, and intentions without the use of spoken or written words. It plays a crucial role in interpersonal interactions and can complement or even override verbal communication. Deep multi-modal learning techniques have shown promise in understanding and analyzing these diverse aspects of BL. The survey emphasizes their applications to BL generation and recognition. Several common BLs are considered i.e., Sign Language (SL), Cued Speech (CS), Co-speech (CoS), and Talking Head (TH), and we have conducted an analysis and established the connections among these four BL for the first time. Their generation and recognition often involve multi-modal approaches. Benchmark datasets for BL research are well collected and organized, along with the evaluation of SOTA methods on these datasets. The survey highlights challenges such as limited labeled data, multi-modal learning, and the need for domain adaptation to generalize models to unseen speakers or languages. Future research directions are presented, including exploring self-supervised learning techniques, integrating contextual information from other modalities, and exploiting large-scale pre-trained multi-modal models. In summary, this survey paper provides a comprehensive understanding of deep multi-modal learning for various BL generations and recognitions for the first time. By analyzing advancements, challenges, and future directions, it serves as a valuable resource for researchers and practitioners in advancing this field. n addition, we maintain a continuously updated paper list for deep multi-modal learning for BL recognition and generation: https://github.com/wentaoL86/awesome-body-language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
