arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.09413cs.SDcs.AIcs.CLcs.MAeess.AS
收藏

Speech-Hands: 一种基于自我反思的语音代理方法用于语音识别和多感知音频推理

Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception

NVIDIA;Kyoto University(京都大学);Carnegie Mellon University(卡内基梅隆大学)
Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hi… 展开作者

Zhen Wan, Chao-Han Huck Yang, Jinchuan Tian, Hanrong Ye, Ankita Pasad, Szu-wei Fu, Arushi Goel, Ryo Hachiuma, Shizhe Diao, Kunal Dhawan, Sreyan Ghosh, Yusuke Hirota, Zhehuai Chen, Rafael Valle, Chenhui Chu, Shinji Watanabe, Yu-Chiang Frank Wang, Boris Ginsburg

AI总结:

本文提出Speech-Hands框架,通过自我反思决策机制解决语音识别和外部声音理解任务中的信任问题,提升了模型在多任务音频推理中的准确性和鲁棒性。

摘要:

我们介绍了一种语音代理框架,该框架学习了一种关键的全方位理解技能:知道何时信任自身,何时咨询外部音频感知。我们的工作受到一个关键但反直觉的发现的启发:简单地在语音识别和外部声音理解任务上微调全方位模型往往会降低性能,因为模型容易被噪声假说误导。为了解决这个问题,我们的框架Speech-Hands将问题重新表述为一个显式的自我反思决策。这个可学习的反思原语在防止模型被错误的外部候选干扰方面证明是有效的。我们展示了这种代理行为机制能够自然地从语音识别推广到复杂的多选音频推理。在OpenASR排行榜上,Speech-Hands在七个基准测试中比强大的基线高出12.1%的WER。该模型在音频问答决策中也实现了77.37%的准确率和高F1分数,展示了在多样化的音频问答数据集上的鲁棒性和可靠性。通过统一感知和决策,我们的工作为更可靠和稳健的音频智能提供了实用路径。

英文摘要:

We introduce a voice-agentic framework that learns one critical omni-understanding skill: knowing when to trust itself versus when to consult external audio perception. Our work is motivated by a crucial yet counterintuitive finding: naively fine-tuning an omni-model on both speech recognition and external sound understanding tasks often degrades performance, as the model can be easily misled by noisy hypotheses. To address this, our framework, Speech-Hands, recasts the problem as an explicit self-reflection decision. This learnable reflection primitive proves effective in preventing the model from being derailed by flawed external candidates. We show that this agentic action mechanism generalizes naturally from speech recognition to complex, multiple-choice audio reasoning. Across the OpenASR leaderboard, Speech-Hands consistently outperforms strong baselines by 12.1% WER on seven benchmarks. The model also achieves 77.37% accuracy and high F1 on audio QA decisions, showing robust generalization and reliability across diverse audio question answering datasets. By unifying perception and decision-making, our work offers a practical path toward more reliable and resilient audio intelligence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑