Gestures perform a variety of communicative functions that powerfully influence human face-to-face interaction. How this communicative function is achieved varies greatly between individuals and depends on the role of the speaker and the context of the interaction. Approaches to automatic gesture generation vary not only in the degree to which they rely on data-driven techniques but also the degree to which they can produce context and speaker specific gestures. However, these approaches face two major challenges: The first is obtaining sufficient training data that is appropriate for the context and the goal of the application. The second is related to designer control to realize their specific intent for the application. Here, we approach these challenges by using large language models (LLMs) to show that these powerful models of large amounts of data can be adapted for gesture analysis and generation. Specifically, we used ChatGPT as a tool for suggesting context-specific gestures that can realize designer intent based on minimal prompts. We also find that ChatGPT can suggests novel yet appropriate gestures not present in the minimal training data. The use of LLMs is a promising avenue for gesture generation that reduce the need for laborious annotations and has the potential to flexibly and quickly adapt to different designer intents.
1D-Touch: NLP-Assisted Coarse Text Selection via a Semi-Direct Gesture
City University of Hong Kong(香港城市大学);University of California San Diego(加利福尼亚大学圣迭戈分校);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
GesGPT: Speech Gesture Synthesis With Text Parsing from ChatGPT
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Engineering Research Center of Mixed Reality and Advanced Display(北京混合现实与先进显示工程研究中心);Institute of Technology, Beijing, China(北京理工大学)
How do Foundation Models Compare to Skeleton-Based Approaches for Gesture Recognition in Human-Robot Interaction?
Chair for Computer Vision, RWTH Aachen University(计算机视觉教研室,亚琛RWTH大学);Robert Bosch GmbH, Corporate Research & Bosch Center for AI(博世公司,博世企业研究院及博世人工智能中心)
GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);The Hong Kong University of Science and Technology(香港科技大学);Goertek Inc.(歌尔股份有限公司);Tsinghua University(清华大学);City University of Hong Kong(香港城市大学)
Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);The Chinese University of Hong Kong(香港中文大学);Huawei Cloud Computing Technologies Co., Ltd(华为云计算技术有限公司)
Who Explains Privacy Policies to Me? Embodied and Textual LLM-Powered Privacy Assistants in Virtual Reality
Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig, Leipzig University(莱比锡大学可扩展数据分析与人工智能中心);LMU Munich(慕尼黑大学);University of the Bundeswehr Munich(联邦国防军慕尼黑大学)
Continuous ErrP detections during multimodal human-robot interaction
German Research Center for Artificial Intelligence (DFKI)(德国人工智能研究中心(DFKI));University of Duisburg-Essen(杜伊斯堡-埃森大学);Institute of Medical Technology Systems(医疗技术系统研究所)