Cued-Agent：一种用于自动提示语识别的协作式多智能体系统
Cued-Agent: A Collaborative Multi-Agent System for Automatic Cued Speech Recognition
AI总结：
针对自动提示语识别中多模态融合数据不足的问题，提出协作式多智能体系统Cued-Agent，通过四个专用子智能体协同实现识别，扩展数据集后在多场景下性能优于现有最优方法。
摘要：
提示语（Cued Speech, CS）是一种结合唇读与手部编码的视觉交流系统，旨在为听障人士的沟通提供便利。自动提示语识别（Automatic CS Recognition, ACSR）旨在通过人工智能驱动的方法，将CS手势与唇部动作转换为文本。传统方法中，手部与唇部动作的时间异步性要求设计复杂模块以实现有效的多模态融合，但受限于数据量不足，现有方法无法充分训练这些融合机制，导致性能欠佳。近年来，多智能体系统在数据有限的复杂任务处理中展现出可观的潜力。为此，我们提出首个面向ACSR的协作式多智能体系统，命名为Cued-Agent。该系统集成了四个专用子智能体：基于多模态大语言模型的手部识别智能体，采用关键帧筛选和CS专家提示策略解码手部动作；基于预训练Transformer的唇部识别智能体，从输入视频中提取唇部特征；手部提示解码智能体，在推理阶段以无需训练的方式动态整合手部提示与唇部特征；自校正音素转单词智能体，首次通过语义优化实现后处理及从音素序列到自然语言句子的端到端转换。为支撑本研究，我们通过采集8名听障提示语使用者的数据，扩展了现有普通话CS数据集，构建了包含14名受试者的混合数据集。大量实验表明，与当前最优方法相比，我们的Cued-Agent在正常场景和听障场景下均表现出色。代码实现可在https://github.com/DennisHgj/Cued-Agent获取。
英文摘要：
Cued Speech (CS) is a visual communication system that combines lip-reading with hand coding to facilitate communication for individuals with hearing impairments. Automatic CS Recognition (ACSR) aims to convert CS hand gestures and lip movements into text via AI-driven methods. Traditionally, the temporal asynchrony between hand and lip movements requires the design of complex modules to facilitate effective multimodal fusion. However, constrained by limited data availability, current methods demonstrate insufficient capacity for adequately training these fusion mechanisms, resulting in suboptimal performance. Recently, multi-agent systems have shown promising capabilities in handling complex tasks with limited data availability. To this end, we propose the first collaborative multi-agent system for ACSR, named Cued-Agent. It integrates four specialized sub-agents: a Multimodal Large Language Model-based Hand Recognition agent that employs keyframe screening and CS expert prompt strategies to decode hand movements, a pretrained Transformer-based Lip Recognition agent that extracts lip features from the input video, a Hand Prompt Decoding agent that dynamically integrates hand prompts with lip features during inference in a training-free manner, and a Self-Correction Phoneme-to-Word agent that enables post-process and end-to-end conversion from phoneme sequences to natural language sentences for the first time through semantic refinement. To support this study, we expand the existing Mandarin CS dataset by collecting data from eight hearing-impaired cuers, establishing a mixed dataset of fourteen subjects. Extensive experiments demonstrate that our Cued-Agent performs superbly in both normal and hearing-impaired scenarios compared with state-of-the-art methods. The implementation is available at https://github.com/DennisHgj/Cued-Agent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
