arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.14647cs.CVcs.RO
收藏

合成无人机基准测试可信赖吗?面向无人机模型选择的真实-合成验证

Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selection

Zhejiang University(浙江大学);Avant Robotics(Avant机器人公司);University of Science and Technology Beijing(北京科技大学);Tsinghua University(清华大学)

Yuyi Zhi, Min Du, Bill Shi, Guodong Yuan, Junheng Liang, Peifan Liao, Xin Guan

AI总结:

该研究针对无人机模型选择,通过两个案例及补充审计发现合成无人机基准的可信度有条件,需针对任务等实证确立有效性,而非仅靠视觉或标签对齐。

摘要:

当特定任务的真实(Real)数据稀缺时,合成(Synthetic)无人机(UAV)图像可扩充部署前的测试证据,但仅靠视觉逼真度或语义标签对齐不足以确立模型选择的有效性。我们在固定评估协议下,通过两个互补案例研究合成评估何时能保留从真实无人机数据得到的模型选择结论。在特定需求的道路 debris 基准中,四个语言条件感知系统在合成数据上的绝对 F1 值更低,但点估计排序和排名第一的模型得以保留;bootstrap 分析未发现支持的成对反转,且大小匹配的重采样保留了排名第一模型的决策。相比之下,在独立实现的业务语料库上进行的闭集压力测试中,使用四个 VisDrone 兼容检测器针对八个经实证支持的类别,改变了点估计排名第一的检测器,并反转了六个成对点估计偏好方向中的三个。对五个无人机业务场景的补充审计进一步揭示,真实-合成在类别构成、目标密度和目标规模上存在显著不匹配,包括小目标的系统性代表性不足。这些结果表明,合成无人机基准的可信度是有条件的:语义覆盖度不足,必须针对感兴趣的任务、运行模式和模型面板实证确立其有效性。

英文摘要:

Synthetic UAV imagery can expand pre-deployment test evidence when task-specific Real data are scarce, but visual plausibility or semantic label alignment alone does not establish validity for model selection. We study when Synthetic evaluation preserves the model-selection conclusions obtained from Real UAV data through two complementary cases under fixed evaluation protocols. In a requirement-specific road-debris benchmark, four language-conditioned perception systems obtain lower absolute F1 on Synthetic data, yet the point-estimate ordering and top-ranked model are preserved; bootstrap analysis yields no supported pairwise reversal, and size-matched resampling preserves the top-model decision. In contrast, a closed-set stress test on a separately materialized business corpus, using four VisDrone-compatible detectors over eight empirically supported classes, changes the point-estimate top-ranked detector and reverses three of six pairwise point-estimate preference directions. A complementary audit of five UAV business scenarios further reveals substantial Real-Synthetic mismatch in class composition, object density, and target scale, including systematic under-representation of small targets. These results show that Synthetic UAV benchmark trust is conditional: semantic coverage is insufficient, and validity must be established empirically for the task, operating regime, and model panel of interest.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑