arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.16364cs.CV

从声音到语义:遥感影像中语音驱动视觉定位的语义监督剖析

From Sound to Sense: An Anatomy of Semantic Supervision for Speech-Driven Visual Grounding in Remote Sensing Imagery

College of Electronic Science, National University of Defense Technology(国防科技大学电子科学学院)

Zhongyu Pang

AI总结:

本研究针对遥感影像语音驱动视觉定位任务,提出SemEcho模型,剖析三种语义注入通道,发布两个音频数据集,发现直接对齐的效果取决于ASR识别器的强弱。

摘要:

遥感影像中的语音驱动视觉定位(RSVG)消除了基于文本交互的输入瓶颈,旨在保留韵律意图,支持时间关键任务中的免提目标获取。直接音频范式将原始语音直接映射到边界框,绕过自动语音识别(ASR),最近被RSTalker采用,但这类定位器实际从语言中学习到的内容尚未被测量。SemEcho将冻结的Whisper语音通路嫁接到VLTVG定位主机上,使语音和文本共享一个解码器,配对文本因该任务的语音语料由DIOR-RSVG表达式合成而天然存在,可作为训练时的教师,以及公平的同主机文本上限。借助该工具,我们通过线性语义探针剖析三种语义注入通道(特征模仿蒸馏、词级CTC监督、两阶段ASR预训练),并发布Audio-DIOR-R(80973个训练片段;10种音色×K=3;Qwen3-TTS合成,在线sox噪声λ∈[0,0.4])和Audio-DIOR-Comp,这是在2×2声学-语义协议下保留17个属性-类别组合的重拆分数据集。我们的剖析得出四项发现:(i)直接对齐是否优于域训练的ASR级联由识别器质量决定:针对弱域内级联(CTC=0.53),直接方法以9.0个Acc@0.5点获胜,转录错误集中在空间关系词上;但针对强域内级联(CTC=0.029),其达到文本上限(80.8 vs.81.0),甚至比最强的直接模型高出8.8个点;反ASR前提仅在对抗弱识别器时成立。

英文摘要:

Speech-driven visual grounding in remote sensing (RSVG) removes the typing bottleneck of text-based interaction and is intended to preserve prosodic intent, enabling hands-free target acquisition in time-critical missions. The direct-audio paradigm, mapping raw speech straight to a bounding box and bypassing automatic speech recognition (ASR), was recently adopted by RSTalker, but what such grounders actually learn about language remains unmeasured. SemEcho grafts a frozen Whisper speech pathway onto a VLTVG grounding host so that speech and text share one decoder, making the paired text, free by construction since speech corpora in this task are synthesized from DIOR-RSVG expressions, a training-time teacher and a fair, same-host text upper bound. On this instrument we dissect three semantic injection channels (feature-mimicry distillation, word-level CTC supervision, and two-stage ASR pretraining) with linear semantic probes, and release Audio-DIOR-R (80,973 training clips; 10 timbres x K=3; Qwen3-TTS synthesis with online sox noise lambda in [0, 0.4]) and Audio-DIOR-Comp, a re-split holding out 17 attribute-category combinations under a 2x2 acoustic-semantic protocol. Our anatomy yields four findings. (i) Whether direct alignment beats a domain-trained ASR cascade is decided by recognizer quality: against a weak in-domain cascade (CTC 0.53) direct wins by 9.0 Acc@0.5 points with transcription errors concentrated on spatial-relation words, but a strong in-domain cascade (CTC 0.029) reaches the text upper bound (80.8 vs. 81.0) and beats even the strongest direct model by 8.8 points; the anti-ASR premise holds only against weak recognizers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑