Structured Prediction with Output Embeddings for Semantic Image Annotation
Xerox Research Europe（施乐欧洲研究中心）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（西班牙国家研究委员会-加泰罗尼亚理工大学工业机器人与信息技术研究所）；TALP Research Center, Universitat Politècnica de Catalunya（加泰罗尼亚理工大学TALP研究中心）
We address the task of annotating images with semantic tuples. Solving this problem requires an algorithm which is able to deal with hundreds of classes for each argument of the tuple. In such contexts, data sparsity becomes a key challenge, as there will be a large number of classes for which only a few examples are available. We propose handling this by incorporating feature representations of both the inputs (images) and outputs (argument classes) into a factorized log-linear model, and exploiting the flexibility of scoring functions based on bilinear forms. Experiments show that integrating feature representations of the outputs in the structured prediction model leads to better overall predictions. We also conclude that the best output representation is specific for each type of argument.
Fine-Grained Rib Fracture Diagnosis with Hyperbolic Embeddings: A Detailed Annotation Framework and Multi-Label Classification Model
Indian Institute of Technology Jodhpur（印度理工学院贾姆pur分校）；All India Institute of Medical Sciences Delhi（全印度医学科学研究所德里）；Government Medical College Srinagar（政府医学院斯里 Nagar）
Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation
Wuhan University of Technology（武汉理工大学）；Huazhong University of Science and Technology（华中科技大学）；Changzhou United lmaging Healthcare Surgical Technology Co. Ltd.（常州联影医疗手术技术有限公司）
提出Mask to Concept (M2C)框架，无需外部模块或重训练，仅用少量标注图像，通过可学习概念嵌入搜索和混合不确定性估计，实现SAM3在医学图像中的自动少样本标注，达到SOTA性能。
Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；ShanghaiTech University（上海科技大学）；The Pennsylvania State University（宾夕法尼亚州立大学）
Recently, there has been growing interest in developing learning-based methods to detect and utilize salient semi-global or global structures, such as junctions, lines, planes, cuboids, smooth surfaces, and all types of symmetries, for 3D scene modeling and understanding. However, the ground truth annotations are often obtained via human labor, which is particularly challenging and inefficient for such tasks due to the large number of 3D structure instances (e.g., line segments) and other factors such as viewpoints and occlusions. In this paper, we present a new synthetic dataset, Structured3D…
Instance object segmentation and tracking provide comprehensive quantification of objects across microscope videos. The recent single-stage pixel-embedding based deep learning approach has shown its superior performance compared with "segment-then-associate" two-stage solutions. However, one major limitation of applying a supervised pixel-embedding based method to microscope videos is the resource-intensive manual labeling, which involves tracing hundreds of overlapped objects with their temporal associations across video frames. Inspired by the recent generative adversarial network (GAN) base…
University of Glasgow（格拉斯哥大学）；TAL Education Group（好未来教育集团）；National University of Singapore（新加坡国立大学）
The current state-of-the-art image-sentence retrieval methods implicitly align the visual-textual fragments, like regions in images and words in sentences, and adopt attention modules to highlight the relevance of cross-modal semantic correspondences. However, the retrieval performance remains unsatisfactory due to a lack of consistent representation in both semantics and structural spaces. In this work, we propose to address the above issue from two aspects: (i) constructing intrinsic structure (along with relations) among the fragments of respective modalities, e.g., "dog $\to$ play $\to$ ba…
University of New South Wales（新南威尔士大学）；University of Technology Sydney（悉尼科技大学）；Raiz Invest Limited（Raiz投资有限公司）；Singapore Management University（新加坡管理大学）
Predicting consumers' purchasing behaviors is critical for targeted advertisement and sales promotion in e-commerce. Human faces are an invaluable source of information for gaining insights into consumer personality and behavioral traits. However, consumer's faces are largely unexplored in previous research, and the existing face-related studies focus on high-level features such as personality traits while neglecting the business significance of learning from facial data. We propose to predict consumers' purchases based on their facial features and purchasing histories. We design a semi-superv…
The need to address the scarcity of task-specific annotated data has resulted in concerted efforts in recent years for specific settings such as zero-shot learning (ZSL) and domain generalization (DG), to separately address the issues of semantic shift and domain shift, respectively. However, real-world applications often do not have constrained settings and necessitate handling unseen classes in unseen domains -- a setting called Zero-shot Domain Generalization, which presents the issues of domain and semantic shifts simultaneously. In this work, we propose a novel approach that learns domain…
Multi-modal word semantics aims to enhance embeddings with perceptual input, assuming that human meaning representation is grounded in sensory experience. Most research focuses on evaluation involving direct visual input, however, visual grounding can contribute to linguistic applications as well. Another motivation for this paper is the growing need for more interpretable models and for evaluating model efficiency regarding size and performance. This work explores the impact of visual information for semantics when the evaluation involves no direct visual input, specifically semantic similari…
GRE-Diff: Gaussian Room Embeddings for Structured Layout Diffusion
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, CSSE, Shenzhen University（广东省视觉媒体与多维智能重点实验室，计算机科学与软件学院，深圳大学）；School of Computer Science, University of Guelph（圭尔夫大学计算机科学学院）
We propose a novel approach to improve a visual-semantic embedding model by incorporating concept representations captured from an external structured knowledge base. We investigate its performance on image classification under both standard and zero-shot settings. We propose two novel evaluation frameworks to analyse classification errors with respect to the class hierarchy indicated by the knowledge base. The approach is tested using the ILSVRC 2012 image dataset and a WordNet knowledge base. With respect to both standard and zero-shot image classification, our approach shows superior perfor…
The University of Western Australia（西澳大利亚大学）；Murdoch University（莫道克大学）；Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology（香港科技大学）
VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning
Beijing University of Posts and Telecommunications（北京邮电大学）；University of Chinese Academy of Sciences（中国科学院大学）；Aerospace Information Research Institute, CAS（中国科学院空天信息创新研究院）；ETH Zurich（苏黎世联邦理工学院）；City University of Hong Kong（香港城市大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Tübingen（图宾根大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
From Drawings to Decisions: A Hybrid Vision-Language Framework for Parsing 2D Engineering Drawings into Structured Manufacturing Knowledge
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR)（新加坡制造技术研究院，新加坡科技研究局）；Advanced Remanufacturing and Technology Centre (ARTC), Agency for Science, Technology and Research (A*STAR)（先进再制造与技术中心，新加坡科技研究局）；Nanyang Technological University（南洋理工大学）
Netaji Subhas Institute of Technology（内塔吉·苏巴斯理工学院）；Netaji Subhas University of Technology（内塔吉·苏巴斯科技大学）
Generative Adversarial Networks (GANs) have revolutionized image synthesis through many applications like face generation, photograph editing, and image super-resolution. Image synthesis using GANs has predominantly been uni-modal, with few approaches that can synthesize images from text or other data modes. Text-to-image synthesis, especially text-to-face synthesis, has promising use cases of robust face-generation from eye witness accounts and augmentation of the reading experience with visual cues. However, only a couple of datasets provide consolidated face data and textual descriptions fo…
