Image annotation aims to annotate a given image with a variable number of class labels corresponding to diverse visual concepts. In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept; 2) how to annotate an image with the optimal number of class labels. To address the first issue, we propose a novel multi-scale deep model for extracting rich and discriminative features capable of representing a wide range of visual concepts. Specifically, a novel two-branch deep neural network architecture is proposed which comprises a very deep main network branch and a companion feature fusion network branch designed for fusing the multi-scale features computed from the main branch. The deep model is also made multi-modal by taking noisy user-provided tags as model input to complement the image input. For tackling the second issue, we introduce a label quantity prediction auxiliary task to the main label prediction task to explicitly estimate the optimal label number for a given image. Extensive experiments are carried out on two large-scale image annotation benchmark datasets and the results show that our method significantly outperforms the state-of-the-art.
UniDiff: Parameter-Efficient Adaptation of Diffusion Models for Land Cover Classification with Multi-Modal Remotely Sensed Imagery and Sparse Annotations
A multi-modal vision-language model for generalizable annotation-free pathology localization
Paul C. Lauterbur Research Center for Biomedical Imaging(生物医学成像研究中心);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Chinese Medicine Guangdong Laboratory(广东中医药实验室);Beijing Chaoyang Hospital, Capital Medical University(首都医科大学北京朝阳医院)
本文提出无标注病理定位视觉-语言模型 AFLoc,通过多层次语义对比学习对齐医学概念与图像特征,在胸部 X 光及多模态数据上超越现有方法并展现强泛化能力。
MultiFly: A Real-World Multimodal Aerial Dataset with Annotation-Efficient Label Transfer and Cross-Modal Semantic Consistency
Fraunhofer Institute IVI(弗劳恩霍夫交通系统研究所);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Cambridge(剑桥大学);Univ. of Applied Sciences Ingolstadt(英戈尔施塔特应用科学大学)
CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images
School of Physics, Peking University(北京大学物理学院);School of Computer Science, Peking University(北京大学计算机学院);Department of Radiotherapy, Peking University Cancer Hospital(北京大学肿瘤医院放疗科);Department of Radiation Oncology, Peking University Third Hospital(北京大学第三医院放疗科)
Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding
Hubei Key Laboratory of Distributed System Security(湖北分布式系统安全重点实验室);Hubei Engineering Research Center on Big Data Security(湖北大数据安全工程研究中心);School of Cyber Science and Engineering(网络安全学院);Huazhong University of Science and Technology(华中科技大学);Peking University(北京大学);Henan University(河南大学);The Chinese University of Hong Kong(香港中文大学);Guangzhou University(广州大学);Protagolabs Inc.(Protagolabs公司)
UEMM-Air: Make Unmanned Aerial Vehicles Perform More Multi-modal Tasks
College of Computer Science and Software Engineering, Hohai University(河海大学计算机与软件学院);College of Artificial Intelligence and Automation, Hohai University(河海大学人工智能与自动化学院);Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);School of Information and Communication Technology, Griffith University(格里菲斯大学信息与通信技术学院)
Partially Supervised Unpaired Multi-Modal Learning for Label-Efficient Medical Image Segmentation
Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(高性能计算研究所(A*STAR));The Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University(山东大学-新加坡南洋理工大学人工智能联合研究中心)
Text-Aided Multi-Modal Panoptic Symbol Spotting for CAD Floor Plan Drawings
CCSE Lab, Beihang University(北京航空航天大学CCSE实验室);School of Transportation Science and Engineering, Beihang University(北京航空航天大学交通科学与工程学院);The Hong Kong Polytechnic University(香港理工大学)
GenCAMO: Scene-Graph Contextual Decoupling for Environment-aware and Mask-free Camouflage Image-Dense Annotation Generation
China University of Petroleum (East China)(中国石油大学(华东));Southwest Jiaotong University(西南交通大学);Beijing Information Science and Technology University(北京信息科技大学);Beihang University(北航)
MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild
School of Computer Science, CUG University of Geosciences (Wuhan)(中国地质大学(武汉)计算机学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Variational Adapter for Cross-modal Similarity Representation
School of Remote Sensing and Information Engineering(遥感与信息工程学院);Wuhan University(武汉大学);School of Data Science and Engineering(数据科学与工程学院);East China Normal University(华东师范大学);State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing(测绘遥感信息工程国家重点实验室)
Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation
The Chinese University of Hong Kong(香港中文大学);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程学系);Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology(广东省计算机视觉与虚拟现实技术重点实验室);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
How Bad is Good enough: Noisy annotations for instrument pose estimation
Technische Universität Darmstadt(达姆施塔特工业大学)
虽然通过深度学习对医学图像的分析在各种应用中取得了前所未有的结果,但很少以系统的方式研究\emph{noisy training annotations}的效果。在医学图像分析中,解决这个问题的大多数报告都集中在研究深度学习分类器的分割性能。在这些研究中缺乏连续的地面实况注释限制了应用结论的价值,其中回归是主要的选择方法。在手术器械姿态估计的应用中,精确度对患者结果有直接的临床影响,研究\emph{noisy annotations}对深度学习姿态估计技术的影响至关重要。由于无法获得地面实况注释,因此真实X射线图像不足以进行该评价。我们规避这个问题,通过生成合成的射线照片,其中地面真实姿态是已知的,因此可以从实验中估计由医学专家的姿态估计误差。此外,该研究显示了深度神经网络在回归设置中从具有足够数据的噪声注释中学习主导信号的特性。
MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations
Shanghai AI Laboratory(上海人工智能实验室);Tsinghua University(清华大学);Shanghai Jiao Tong University(上海交通大学);Zhejiang University(浙江大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);Zhiyuan College, Shanghai Jiao Tong University(上海交通大学紫阳学院);CPII under InnoHK(创新香港下的CPII)
Cranio-ID: Graph-Based Craniofacial Identification via Automatic Landmark Annotation in 2D Multi-View X-rays
Visual Intelligence and Machine Learning (VIML) Group, School of Computing and Electrical Engineering, Indian Institute of Technology Mandi(视觉智能与机器学习(VIML)小组,计算与电子工程学院,印度理工学院曼迪)