面向使用头部运动定位对话伙伴
Towards Localizing Conversation Partners using Head Motion
AI总结:
本文提出通过头部运动行为定位对话伙伴的声学兴趣区域,引入HALo网络和CoCo算法,在嘈杂环境中提升对话识别性能。
摘要:
许多人在嘈杂环境中难以理解对话伙伴,尤其是背景说话者或听力障碍者。新兴的智能眼镜等可穿戴设备为增强对话语音提供了变革性机会。关键在于识别用户希望倾听的方向,即用户的声学兴趣区域。尽管现有基于空间音频的方法可以确定语音输入方向,但它们对听觉偏好不敏感且在有干扰说话者的情况下功能有限。为此,需要行为线索来主动推断用户的声学兴趣区域。我们探讨了通过智能眼镜上的惯性测量单元(IMUs)捕捉的头部定向行为作为模态,用于定位坐姿对话中的这些区域。我们引入HALo,一种基于头部方向的声学区域定位网络,利用智能眼镜的IMUs非侵入性地推断与对话伙伴位置对应的听觉区域。通过整合事先估计的对话伙伴数量,我们的方法在现有方法上实现了21%的性能提升。我们还补充了CoCo,仅使用IMU数据分类对话伙伴数量,达到0.74的准确率,并在规则基和通用时间序列基线中实现了35%的提升。我们讨论了特征提取和推断的实践考虑,并提供了长时间段的定性分析。我们还展示了一个最小的端到端语音增强系统,表明基于头部方向的定位在多个对话伙伴的极端嘈杂环境中具有明显优势。
英文摘要:
Many individuals struggle to understand conversation partners in noisy settings, particularly amid background speakers or due to hearing impairments. Emerging wearables like smartglasses offer a transformative opportunity to enhance speech from conversation partners. Crucial to this is identifying the direction in which the user wants to listen, which we refer to as the user's acoustic zones of interest. While current spatial audio-based methods can resolve the direction of vocal input, they are agnostic to listening preferences and have limited functionality in noisy settings with interfering speakers. To address this, behavioral cues are needed to actively infer a user's acoustic zones of interest. We explore the effectiveness of head-orienting behavior, captured by Inertial Measurement Units (IMUs) on smartglasses, as a modality for localizing these zones in seated conversations. We introduce HALo, a head-orientation-based acoustic zone localization network that leverages smartglasses' IMUs to non-invasively infer auditory zones of interest corresponding to conversation partner locations. By integrating an a priori estimate of the number of conversation partners, our approach yields a 21% performance improvement over existing methods. We complement this with CoCo, which classifies the number of conversation partners using only IMU data, achieving 0.74 accuracy and a 35% gain over rule-based and generic time-series baselines. We discuss practical considerations for feature extraction and inference and provide qualitative analyses over extended sessions. We also demonstrate a minimal end-to-end speech enhancement system, showing that head-orientation-based localization offers clear advantages in extremely noisy settings with multiple conversation partners.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。