We propose an end-to-end network for image generation from given structured-text that consists of the visual-relation layout module and the pyramid of GANs, namely stacking-GANs. Our visual-relation layout module uses relations among entities in the structured-text in two ways: comprehensive usage and individual usage. We comprehensively use all available relations together to localize initial bounding-boxes of all the entities. We also use individual relation separately to predict from the initial bounding-boxes relation-units for all the relations in the input text. We then unify all the relation-units to produce the visual-relation layout, i.e., bounding-boxes for all the entities so that each of them uniquely corresponds to each entity while keeping its involved relations. Our visual-relation layout reflects the scene structure given in the input text. The stacking-GANs is the stack of three GANs conditioned on the visual-relation layout and the output of previous GAN, consistently capturing the scene structure. Our network realistically renders entities' details in high resolution while keeping the scene structure. Experimental results on two public datasets show outperformances of our method against state-of-the-art methods.
Modular Diffusion Models for Structured Visual Recognition
Department of Computer Science, University of British Columbia（不列颠哥伦比亚大学计算机科学系）；Vector Institute for AI（向量人工智能研究所）；CompVis, Ludwig Maximilian University of Munich（慕尼黑路德维希·马克西米利安大学计算机视觉实验室）；Munich Center for Machine Learning（慕尼黑机器学习中心）；CIFAR AI Chair（CIFAR人工智能教席）
Factuality Matters: When Image Generation and Editing Meet Structured Visuals
CUHK MMLab（香港大学多模态实验室）；Beihang University（北京航空航天大学）；Krea AI（Krea人工智能）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Lab（上海人工智能实验室）；Hugging Face；National University of Singapore（新加坡国立大学）；ByteDance（字节跳动）；The University of Hong Kong（香港大学）
StruVis: Enhancing Reasoning-based Text-to-Image Generation via Thinking with Structured Vision
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Ant Group（蚂蚁集团）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology（香港科技大学）；East China Normal University（华东师范大学）
Visual Question Answering (VQA) attracts much attention from both industry and academia. As a multi-modality task, it is challenging since it requires not only visual and textual understanding, but also the ability to align cross-modality representations. Previous approaches extensively employ entity-level alignments, such as the correlations between the visual regions and their semantic labels, or the interactions across question words and object features. These attempts aim to improve the cross-modality representations, while ignoring their internal relations. Instead, we propose to apply st…
The University of Western Australia（西澳大利亚大学）；Murdoch University（莫道克大学）；Sun Yat-sen University（中山大学）；The Hong Kong University of Science and Technology（香港科技大学）
Structured representations, such as Bags of Words, VLAD and Fisher Vectors, have proven highly effective to tackle complex visual recognition tasks. As such, they have recently been incorporated into deep architectures. However, while effective, the resulting deep structured representation learning strategies typically aggregate local features from the entire image, ignoring the fact that, in complex recognition tasks, some regions provide much more discriminative information than others. In this paper, we introduce an attentional structured representation learning framework that incorporates …
Visual attention, which assigns weights to image regions according to their relevance to a question, is considered as an indispensable part by most Visual Question Answering models. Although the questions may involve complex relations among multiple regions, few attention models can effectively encode such cross-region relations. In this paper, we demonstrate the importance of encoding such relations by showing the limited effective receptive field of ResNet on two datasets, and propose to model the visual attention as a multivariate distribution over a grid-structured Conditional Random Field…
Tsinghua University（清华大学）；Virginia Tech（弗吉尼亚理工大学）；University of California, Merced（加利福尼亚大学默塞德分校）
Numerous embedding models have been recently explored to incorporate semantic knowledge into visual recognition. Existing methods typically focus on minimizing the distance between the corresponding images and texts in the embedding space but do not explicitly optimize the underlying structure. Our key observation is that modeling the pairwise image-image relationship improves the discrimination ability of the embedding model. In this paper, we propose the structured discriminative and difference constraints to learn visual-semantic embeddings. First, we exploit the discriminative constraints …
AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning
Research Center for Space Computing System, Zhejiang Lab, Hangzhou（杭州浙大实验室空间计算系统研究中心）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou（中国科学院大学杭州高等研究院）；School of Cyber Science and Engineering, Zhengzhou University（郑州大学计算机科学与工程学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）
QUvA Lab, University of Amsterdam（阿姆斯特丹大学QUvA实验室）；University of Amsterdam（阿姆斯特丹大学）
This paper studies visual search using structured queries. The structure is in the form of a 2D composition that encodes the position and the category of the objects. The transformation of the position and the category of the objects leads to a continuous-valued relationship between visual compositions, which carries highly beneficial information, although not leveraged by previous techniques. To that end, in this work, our goal is to leverage these continuous relationships by using the notion of symmetry in equivariance. Our model output is trained to change symmetrically with respect to the …
Visual data such as images and videos contain a rich source of structured semantic labels as well as a wide range of interacting components. Visual content could be assigned with fine-grained labels describing major components, coarse-grained labels depicting high level abstractions, or a set of labels revealing attributes. Such categorization over different, interacting layers of labels evinces the potential for a graph-based encoding of label information. In this paper, we exploit this rich structure for performing graph-based inference in label space for a number of tasks: multi-label image…
Visual question answering (VQA) is of significant interest due to its potential to be a strong test of image understanding systems and to probe the connection between language and vision. Despite much recent progress, general VQA is far from a solved problem. In this paper, we focus on the VQA multiple-choice task, and provide some good practices for designing an effective VQA model that can capture language-vision interactions and perform joint reasoning. We explore mechanisms of incorporating part-of-speech (POS) tag guided attention, convolutional n-grams, triplet attention interactions bet…
Temporal Evidence Routing with Structured Visual Evidence for TimeLogicQA
Southeast University（东南大学）；National University of Singapore（新加坡国立大学）；Independent Researcher（独立研究员）；Opus AI Research（Opus AI研究院）；University of Science and Technology of China（中国科学技术大学）
Xidian University（西安电子科技大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）
Visual Emotion Analysis (VEA) has attracted increasing attention recently with the prevalence of sharing images on social networks. Since human emotions are ambiguous and subjective, it is more reasonable to address VEA in a label distribution learning (LDL) paradigm rather than a single-label classification task. Different from other LDL tasks, there exist intrinsic relationships between emotions and unique characteristics within them, as demonstrated in psychological theories. Inspired by this, we propose a well-grounded circular-structured representation to utilize the prior knowledge for v…
University of Glasgow（格拉斯哥大学）；TAL Education Group（好未来教育集团）；National University of Singapore（新加坡国立大学）
The current state-of-the-art image-sentence retrieval methods implicitly align the visual-textual fragments, like regions in images and words in sentences, and adopt attention modules to highlight the relevance of cross-modal semantic correspondences. However, the retrieval performance remains unsatisfactory due to a lack of consistent representation in both semantics and structural spaces. In this work, we propose to address the above issue from two aspects: (i) constructing intrinsic structure (along with relations) among the fragments of respective modalities, e.g., "dog $\to$ play $\to$ ba…
ViewSRD: 3D Visual Grounding via Structured Multi-View Decomposition
South China University of Technology（华南理工大学）；Guangdong Engineering Center for Large Model and GenAI Technology（广东省大模型与生成式人工智能技术工程中心）；State Key Laboratory of Subtropical Building and Urban Science（亚热带建筑科学国家重点实验室）；Ministry of Education Key Laboratory of Big Data and Intelligent Robot（教育部大数据与智能机器人重点实验室）；Guangdong Provincial Key Lab of Computational Intelligence and Cyberspace Information（广东省计算智能与网络信息重点实验室）；Singapore Management University（新加坡国立大学）
Beyond Classification: Structured Supervision Aligns Visual Evidence with Medical Semantics
Zhejiang University（浙江大学）；The Second Affiliated Hospital Zhejiang University（浙江大学医学院附属第二医院）；Transvascular Implantation Devices Research Institute and State Key Laboratory of Transvascular Implantation Devices（经血管植入器械研究院及经血管植入器械全国重点实验室）；Henan Normal University（河南师范大学）；University of Southampton（南安普顿大学）；School of Software Technology Zhejiang University（浙江大学软件学院）
Harbin Institute of Technology（哈尔滨工业大学）；INRIA（法国国家信息与自动化研究所）；Hong Kong University of Science and Technology（香港科技大学）；University of Trento（特伦托大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
Convolutional neural networks have enabled major progresses in addressing pixel-level prediction tasks such as semantic segmentation, depth estimation, surface normal prediction and so on, benefiting from their powerful capabilities in visual representation learning. Typically, state of the art models integrate attention mechanisms for improved deep feature representations. Recently, some works have demonstrated the significance of learning and combining both spatial- and channelwise attentions for deep feature refinement. In this paper, weaim at effectively boosting previous approaches and pr…
Structured Pruning for Efficient Visual Place Recognition
School of Electronics and Computer Science, University of Southampton（南安普顿大学电子与计算机科学学院）；School of Electrical Engineering and Computer Science, Queensland University of Technology（昆士兰科技大学电气工程与计算机科学学院）；School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）
National Chiao Tung University（国立交通大学）；National University of Singapore（新加坡国立大学）
As the intermediate-level representations bridging the two levels, structured representations of visual scenes, such as visual relationships between pairwise objects, have been shown to not only benefit compositional models in learning to reason along with the structures but provide higher interpretability for model decisions. Nevertheless, these representations receive much less attention than traditional recognition tasks, leaving numerous open challenges unsolved. In the thesis, we study how machines can describe the content of the individual image or video with visual relationships as the …
Img2CAD: Conditioned 3D CAD Model Generation from Single Image with Structured Visual Geometry
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；KOKONI3D, Moxin (Huzhou) Technology Co., LTD（墨芯（湖州）科技有限公司）；School of Software Technology, Zhejiang University（浙江大学软件学院）；School of Information Engineering, Huzhou University（湖州师范学院信息工程学院）
Bar Ilan University（巴伊兰大学）；Trax Image Recognition（Trax图像识别公司）
This paper presents a novel deep learning architecture to classify structured objects in datasets with a large number of visually similar categories. We model sequences of images as linear-chain CRFs, and jointly learn the parameters from both local-visual features and neighboring classes. The visual features are computed by convolutional layers, and the class embeddings are learned by factorizing the CRF pairwise potential matrix. This forms a highly nonlinear objective function which is trained by optimizing a local likelihood approximation with batch-normalization. This model overcomes the …
Structured Visual Content (SVC) such as graphs, flow charts, or the like are used by authors to illustrate various concepts. While such depictions allow the average reader to better understand the contents, images containing SVCs are typically not machine-readable. This, in turn, not only hinders automated knowledge aggregation, but also the perception of displayed in-formation for visually impaired people. In this work, we propose a synthetic dataset, containing SVCs in the form of images as well as ground truths. We show the usage of this dataset by an application that automatically extracts…
Multi-modal word semantics aims to enhance embeddings with perceptual input, assuming that human meaning representation is grounded in sensory experience. Most research focuses on evaluation involving direct visual input, however, visual grounding can contribute to linguistic applications as well. Another motivation for this paper is the growing need for more interpretable models and for evaluating model efficiency regarding size and performance. This work explores the impact of visual information for semantics when the evaluation involves no direct visual input, specifically semantic similari…
We propose a novel approach to improve a visual-semantic embedding model by incorporating concept representations captured from an external structured knowledge base. We investigate its performance on image classification under both standard and zero-shot settings. We propose two novel evaluation frameworks to analyse classification errors with respect to the class hierarchy indicated by the knowledge base. The approach is tested using the ILSVRC 2012 image dataset and a WordNet knowledge base. With respect to both standard and zero-shot image classification, our approach shows superior perfor…
Motivated by the application of fact-level image understanding, we present an automatic method for data collection of structured visual facts from images with captions. Example structured facts include attributed objects (e.g., <flower, red>), actions (e.g., <baby, smile>), interactions (e.g., <man, walking, dog>), and positional information (e.g., <vase, on, table>). The collected annotations are in the form of fact-image pairs (e.g.,<man, walking, dog> and an image region containing this fact). With a language approach, the proposed method is able to collect hundreds of thousands of visual f…
University of Amsterdam（阿姆斯特丹大学）；Priberam Labs（Priberam实验室）；Institute of Systems and Robotics（系统与机器人研究所）；LUMLIS (Lisbon ELLIS Unit)（LUMLIS（里斯本ELLIS单元））；Unbabel（Unbabel公司）
Visual attention mechanisms are widely used in multimodal tasks, as visual question answering (VQA). One drawback of softmax-based attention mechanisms is that they assign some probability mass to all image regions, regardless of their adjacency structure and of their relevance to the text. In this paper, to better link the image structure with the text, we replace the traditional softmax attention mechanism with two alternative sparsity-promoting transformations: sparsemax, which is able to select only the relevant regions (assigning zero weight to the rest), and a newly proposed Total-Variat…
Humans are highly efficient learners, with the ability to grasp the meaning of a new concept from just a few examples. Unlike popular computer vision systems, humans can flexibly leverage the compositional structure of the visual world, understanding new concepts as combinations of existing concepts. In the current paper, we study how people learn different types of visual compositions, using abstract visual forms with rich relational structure. We find that people can make meaningful compositional generalizations from just a few examples in a variety of scenarios, and we develop a Bayesian pr…
Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning
School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）；School of Artificial Intelligence, Shenzhen University（深圳大学人工智能学院）；Graduate School of Advanced Science and Engineering, Hiroshima University（广岛大学先进科学与工程研究生院）
We cast visual imitation as a visual correspondence problem. Our robotic agent is rewarded when its actions result in better matching of relative spatial configurations for corresponding visual entities detected in its workspace and teacher's demonstration. We build upon recent advances in Computer Vision,such as human finger keypoint detectors, object detectors trained on-the-fly with synthetic augmentations, and point detectors supervised by viewpoint changes and learn multiple visual entity detectors for each demonstration without human annotations or robot interactions. We empirically show…
