arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.22131cs.HC
收藏

为动作赋予意义:通过将无辅助AAC与语音生成信息结合扩展沟通的挑战与机遇

Giving Meaning to Movements: Challenges and Opportunities in Expanding Communication by Pairing Unaided AAC with Speech Generated Messages

Pennsylvania State University(宾夕法尼亚州立大学);See CVI, Speak AAC;L. L. SLIM, LLC.;Florida State University(佛罗里达州立大学)

Imran Kabir, Sharon Ann Redmon, Lynn R Elko, Kevin Williams, Mitchell A Case, Dawn J Sowers, Krista Wilkinson, Syed Masum Billah

AI总结:

本文提出AllyAAC系统,通过结合无辅助AAC与语音生成信息,解决个性化肢体动作识别挑战,提升沟通效率与可理解性。

摘要:

增强型与替代型交流(AAC)技术可分为两种形式:有辅助的AAC,它使用外部设备如语音生成系统来产生标准化输出;无辅助的AAC则依赖于基于身体的肢体动作进行自然表达,但需要共享的理解。我们研究如何将这些方法结合,以利用无辅助AAC的速度和自然性,同时保持有辅助AAC的可理解性,这对有交流和运动障碍的人而言是一个鲜为人知的领域。通过18个月的参与设计与AAC用户合作,我们识别出关键挑战和机遇,并开发了AllyAAC,一种可穿戴系统,其腕部惯性测量单元(IMU)与智能手机应用相结合。我们通过与14名参与者进行实地研究评估了AllyAAC,并生成了一个包含超过60万个多模态数据点的数据集,其中包括非典型的肢体动作——这是首次此类数据集。我们的发现揭示了在识别个性化、异步肢体动作方面的挑战,并展示了如何使用基于Transformer的大型机器学习(ML)模型和不同的预训练策略来解决这些问题。总之,我们为结合有辅助和无辅助AAC的自适应、个性化系统贡献了设计原则和参考实现。

英文摘要:

Augmentative and Alternative Communication (AAC) technologies are categorized into two forms: aided AAC, which uses external devices like speech-generating systems to produce standardized output, and unaided AAC, which relies on body-based gestures for natural expression but requires shared understanding. We investigate how to combine these approaches to harness the speed and naturalness of unaided AAC while maintaining the intelligibility of aided AAC, a largely unexplored area for individuals with communication and motor impairments. Through 18 months of participatory design with AAC users, we identified key challenges and opportunities and developed AllyAAC, a wearable system with a wrist-worn IMU paired with a smartphone app. We evaluated AllyAAC in a field study with 14 participants and produced a dataset containing over 600,000 multimodal data points featuring atypical gestures--the first of its kind. Our findings reveal challenges in recognizing personalized, idiosyncratic gestures and demonstrate how to address them using Transformer-based large machine learning (ML) models with different pretraining strategies. In sum, we contribute design principles and a reference implementation for adaptive, personalized systems combining aided and unaided AAC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑