arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.00652cs.CV
收藏

OTTER:面向多模态理解的基于文本-图像表征的开放标注框架

OTTER: Open-Tagging via Text-Image Representation for Multi-modal Understanding

Huawei Singapore Research Center(华为新加坡研究中心)

Jieer Ouyang, Xiaoneng Xiang, Zheng Wang, Yangkai Ding

AI总结:

本研究提出多模态开放标注框架OTTER,基于层级多模态数据集与多头注意力架构对齐图文表征和两类标签嵌入,在两个基准上显著优于基线,兼顾闭集标注稳定性与开放标签适应性。

摘要:

我们提出了OTTER,这是一个统一的开放集多标签标注框架,兼顾了经整理的预定义类别集的稳定性与用户驱动的开放标签的适应性。OTTER基于一个大规模、层级化组织的多模态数据集构建,该数据集从各类在线资源库收集,通过结合自动化视觉-语言标注与人工优化的混合流水线完成标注。借助多头注意力架构,OTTER将视觉与文本表征同时与固定标签嵌入和开放集标签嵌入对齐,实现动态且语义一致的标注。OTTER在两个基准数据集上始终优于同类基线模型:在Otter数据集上整体F1得分为0.81,在Favorite数据集上为0.75,分别比次优结果高出0.10和0.02。OTTER在开放集标签上取得近乎完美的表现,在Otter数据集上F1为0.99,在Favorite数据集上为0.97,同时在预定义标签上保持了有竞争力的准确率。这些结果证明,OTTER在多模态标注应用中有效衔接了闭集一致性与开放词汇灵活性。

英文摘要:

We introduce OTTER, a unified open-set multi-label tagging framework that harmonizes the stability of a curated, predefined category set with the adaptability of user-driven open tags. OTTER is built upon a large-scale, hierarchically organized multi-modal dataset, collected from diverse online repositories and annotated through a hybrid pipeline combining automated vision-language labeling with human refinement. By leveraging a multi-head attention architecture, OTTER jointly aligns visual and textual representations with both fixed and open-set label embeddings, enabling dynamic and semantically consistent tagging. OTTER consistently outperforms competitive baselines on two benchmark datasets: it achieves an overall F1 score of 0.81 on Otter and 0.75 on Favorite, surpassing the next-best results by margins of 0.10 and 0.02, respectively. OTTER attains near-perfect performance on open-set labels, with F1 of 0.99 on Otter and 0.97 on Favorite, while maintaining competitive accuracy on predefined labels. These results demonstrate OTTER's effectiveness in bridging closed-set consistency with open-vocabulary flexibility for multi-modal tagging applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑