Given a question-image input, the Visual Commonsense Reasoning (VCR) model can predict an answer with the corresponding rationale, which requires inference ability from the real world. The VCR task, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge, is a cognition-level scene understanding task. The VCR task has aroused researchers' interest due to its wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and losing information in long sequences. In this paper, we propose a parallel attention-based cognitive VCR network PAVCR, which fuses visual-textual information efficiently and encodes semantic information in parallel to enable the model to capture rich information for cognition-level inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning.
Cognition-Oriented Emotion Tracing from Causes to Consequences in Real-World Social Scenes
National University of Singapore（新加坡国立大学）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；University of Oxford（牛津大学）；Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算与智能研究院）
Agentic Collaborative Cognition for Zero-Shot 3D Understanding
Sichuan University（四川大学）；Adelaide University（阿德莱德大学）；University of Electronic Science and Technology of China（电子科技大学）；Beijing Institute of Technology（北京理工大学）
DM-KG: A Novel Method for Boosting Spatial Cognition of Vision-Language Models in Street View Imagery
Institute of Surveying and Mapping, Information Engineering University（信息工程大学测绘学院）；Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences（中国科学院地理科学与资源研究所）
Image captioning has attracted ever-increasing research attention in the multimedia community. To this end, most cutting-edge works rely on an encoder-decoder framework with attention mechanisms, which have achieved remarkable progress. However, such a framework does not consider scene concepts to attend visual information, which leads to sentence bias in caption generation and defects the performance correspondingly. We argue that such scene concepts capture higher-level visual semantics and serve as an important cue in describing images. In this paper, we propose a novel scene-based factored…
CVSBench: A Comprehensive Benchmark for Cross-view Spatial Reasoning and Dreaming
Faculty of Electronic and Information Engineering, Xi’an Jiaotong University（西安交通大学电子与信息工程学院）；Ministry of Education Key Laboratory of Intelligent Networks and Network Security（教育部智能网络与网络安全重点实验室）
The Pennsylvania State University（宾夕法尼亚州立大学）；Google Inc.（谷歌公司）
Many tasks are related to determining if a particular text string exists in an image. In this work, we propose a new framework that learns this task in an end-to-end way. The framework takes an image and a text string as input and then outputs the probability of the text string being present in the image. This is the first end-to-end framework that learns such relationships between text and images in scene text area. The framework does not require explicit scene text detection or recognition and thus no bounding box annotations are needed for it. It is also the first work in scene text area th…
Sony Interactive Entertainment LLC（索尼互动娱乐有限责任公司）；Georgia Institute of Technology（佐治亚理工学院）；State University of New York at Binghamton（纽约州立大学宾汉姆顿分校）
Image captioning has attracted considerable attention in recent years. However, little work has been done for game image captioning which has some unique characteristics and requirements. In this work we propose a novel game image captioning model which integrates bottom-up attention with a new multi-level residual top-down attention mechanism. Firstly, a lower-level residual top-down attention network is added to the Faster R-CNN based bottom-up attention network to address the problem that the latter may lose important spatial information when extracting regional features. Secondly, an upper…
In this paper, we present a novel approach for the task of eXplainable Question Answering (XQA), i.e., generating natural language (NL) explanations for the Visual Question Answering (VQA) problem. We generate NL explanations comprising of the evidence to support the answer to a question asked to an image using two sources of information: (a) annotations of entities in an image (e.g., object labels, region descriptions, relation phrases) generated from the scene graph of the image, and (b) the attention map generated by a VQA model when answering the question. We show how combining the visual …
CogToM: A Comprehensive Theory of Mind Benchmark inspired by Human Cognition for Large Language Models
BrainCog Lab, Institute of Automation, Chinese Academy of Sciences（脑认知实验室，自动化研究所，中国科学院）；Beijing Institute of AI Safety and Governance (Beijing-AISI)（北京人工智能安全与治理研究所）；Beijing Key Laboratory of Safe AI and Superalignment（北京安全人工智能与超对齐重点实验室）；School of Artificial Intelligence, UCAS（人工智能学院，中国科学院大学）；Long-term AI（长期人工智能）
School of Mathematics, Tianjin University（天津大学数学学院）；School of Computing, Merchiston Campus, Edinburgh Napier University（爱丁堡龙比亚大学计算学院）
Segmentation-based methods are widely used for scene text detection due to their superiority in describing arbitrary-shaped text instances. However, two major problems still exist: 1) current label generation techniques are mostly empirical and lack theoretical support, discouraging elaborate label design; 2) as a result, most methods rely heavily on text kernel segmentation which is unstable and requires deliberate tuning. To address these challenges, we propose a human cognition-inspired framework, termed, Conceptual Text Region Network (CTRNet). The framework utilizes Conceptual Text Region…
It is almost universal to regard attention as the facility that permits an agent, human or machine, to give priority processing resources to relevant stimuli while ignoring the irrelevant. The reality of how this might manifest itself throughout all the forms of perceptual and cognitive processes possessed by humans, however, is not as clear. Here we examine this reality with a broad perspective in order to highlight the myriad ways that attentional processes impact both perception and cognition. The paper concludes by showing two real world problems that exhibit sufficient complexity to illus…
Indian Institute of Technology, Madras（印度理工学院马德拉斯分校）；Queen Marys College, Chennai（钦奈玛丽皇后学院）；National Institute of Technology（国家理工学院）
There is amazing progress in Deep Learning based models for Image captioning and Low Light image enhancement. For the first time in literature, this paper develops a Deep Learning model that translates night scenes to sentences, opening new possibilities for AI applications in the safety of visually impaired women. Inspired by Image Captioning and Visual Question Answering, a novel Interactive Image Captioning is developed. A user can make the AI focus on any chosen person of interest by influencing the attention scoring. Attention context vectors are computed from CNN feature vectors and user…
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Peking University（北京大学）；University of Adelaide（阿德莱德大学）；Beihang University（北京航空航天大学）
Scene graphs are semantic abstraction of images that encourage visual understanding and reasoning. However, the performance of Scene Graph Generation (SGG) is unsatisfactory when faced with biased data in real-world scenarios. Conventional debiasing research mainly studies from the view of balancing data distribution or learning unbiased models and representations, ignoring the correlations among the biased classes. In this work, we analyze this problem from a novel cognition perspective: automatically building a hierarchical cognitive structure from the biased predictions and navigating that …
MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Tencent（腾讯）；Nankai University（南开大学）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University of Science and Technology（南京理工大学）
National University of Singapore（新加坡国立大学）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究院）
Recent work in cognitive reasoning and computer vision has engendered an increasing popularity for the Violation-of-Expectation (VoE) paradigm in synthetic datasets. Inspired by work in infant psychology, researchers have started evaluating a model's ability to discriminate between expected and surprising scenes as a sign of its reasoning ability. Existing VoE-based 3D datasets in physical reasoning only provide vision data. However, current cognitive models of physical reasoning by psychologists reveal infants create high-level abstract representations of objects and interactions. Capitalizin…
Learning to Look before Learning to Like: Incorporating Human Visual Cognition into Aesthetic Quality Assessment
Faculty of Data Science, City University of Macau（澳门城市大学数据科学学院）；Zhongshan Ophthalmic Center, Sun Yat-sen University（中山眼科中心，中山大学）；School of computer science, Torrens University Australia（计算机科学学院，托伦斯大学澳大利亚分校）
Following the Diagnostic Trace: Visual Cognition-guided Cooperative Network for Chest X-Ray Diagnosis
College of Computer Science, Northwest University（西北大学计算机学院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（西南交通大学计算机与人工智能学院）；Department of PET/CT, The First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院PET/CT科）
We position a narrative-centred computational model for high-level knowledge representation and reasoning in the context of a range of assistive technologies concerned with "visuo-spatial perception and cognition" tasks. Our proposed narrative model encompasses aspects such as \emph{space, events, actions, change, and interaction} from the viewpoint of commonsense reasoning and learning in large-scale cognitive systems. The broad focus of this paper is on the domain of "human-activity interpretation" in smart environments, ambient intelligence etc. In the backdrop of a "smart meeting cinematog…
Joint attention is a core, early-developing form of social interaction. It is based on our ability to discriminate the third party objects that other people are looking at. While it has been shown that people can accurately determine whether another person is looking directly at them versus away, little is known about human ability to discriminate a third person gaze directed towards objects that are further away, especially in unconstraint cases where the looker can move her head and eyes freely. In this paper we address this question by jointly exploring human psychophysics and a cognitively…
Subjective Camera 1.0: Bridging Human Cognition and Visual Reconstruction through Sequence-Aware Sketch-Guided Diffusion
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University（国家多媒体软件工程研究中心，人工智能研究院，计算机科学学院，武汉大学）；Hubei Key Laboratory of Multimedia and Network Communication Engineering（湖北省多媒体与网络通信工程重点实验室）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）；CyberAgent AI Lab, Japan（CyberAgent AI实验室，日本）
本文提出Subjective Camera 1.0框架，通过文本描述和逐步草图等主观读数，利用序列感知草图引导扩散模型重建真实场景，无需大规模配对训练数据，在图像质量与空间语义对齐上达到最先进水平。
UAV traffic scene understanding: A regulation embedded multi-modal network and a unified benchmark
Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China（中国科学院计算机网络信息中心，北京100190，中国）；University of Chinese Academy of Sciences, Beijing 100190, China（中国科学院大学，北京100190，中国）；Anhui Provincial Key Laboratory of Multi-modal Cognitive Computation, Anhui University, Hefei 230601, China（安徽省多模态认知计算重点实验室，安徽大学，合肥230601，中国）；Information Materials and Intelligent Sensing Laboratory of Anhui Province, Anhui University, Hefei 230601, China（安徽省信息材料与智能感知实验室，安徽大学，合肥230601，中国）
University of Kansas（堪萨斯大学）；Sony Interactive Entertainment Global R&D（索尼互动娱乐全球研发部）；Ryerson University（瑞尔森大学）
The paper proposes a new text recognition network for scene-text images. Many state-of-the-art methods employ the attention mechanism either in the text encoder or decoder for the text alignment. Although the encoder-based attention yields promising results, these schemes inherit noticeable limitations. They perform the feature extraction (FE) and visual attention (VA) sequentially, which bounds the attention mechanism to rely only on the FE final single-scale output. Moreover, the utilization of the attention process is limited by only applying it directly to the single scale feature-maps. To…
Shanghai Jiao Tong University（上海交通大学）；Westone Cryptologic Research Center（卫士通密码研究中心）
In this paper, we propose Double Supervised Network with Attention Mechanism (DSAN), a novel end-to-end trainable framework for scene text recognition. It incorporates one text attention module during feature extraction which enforces the model to focus on text regions and the whole framework is supervised by two branches. One supervision branch comes from context-level modelling and another comes from one extra supervision enhancement branch which aims at tackling inexplicit semantic information at character level. These two supervisions can benefit each other and yield better performance. Th…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Visual understanding goes well beyond object recognition. With one glance at an image, we can effortlessly imagine the world beyond the pixels: for instance, we can infer people's actions, goals, and mental states. While this task is easy for humans, it is tremendously difficult for today's vision systems, requiring higher-order cognition and commonsense reasoning about the world. We formalize this task as Visual Commonsense Reasoning. Given a challenging question about an image, a machine must answer correctly and then provide a rationale justifying its answer. Next, we introduce a new datase…
Xidian University（西安电子科技大学）；University of Montenegro（黑山大学）
Despite the tremendous success of convolutional neural networks (CNNs) in computer vision, the mechanism of CNNs still lacks clear interpretation. Currently, class activation mapping (CAM), a famous visualization technique to interpret CNN's decision, has drawn increasing attention. Gradient-based CAMs are efficient while the performance is heavily affected by gradient vanishing and exploding. In contrast, gradient-free CAMs can avoid computing gradients to produce more understandable results. However, existing gradient-free CAMs are quite time-consuming because hundreds of forward interferenc…
This paper addresses the challenging problem of estimating the general visual attention of people in images. Our proposed method is designed to work across multiple naturalistic social scenarios and provides a full picture of the subject's attention and gaze. In contrast, earlier works on gaze and attention estimation have focused on constrained problems in more specific contexts. In particular, our model explicitly represents the gaze direction and handles out-of-frame gaze targets. We leverage three different datasets using a multi-task learning approach. We evaluate our method on widely use…
Attention over Scene Graphs: Indoor Scene Representations Toward CSAI Classification
Instituto de Computação (IC) Universidade Estadual de Campinas (UNICAMP)（计算机学院（IC）Campinas大学（UNICAMP））；Departamento de Ciência da Computação (DCC) Universidade Federal de Minas Gerais (UFMG)（计算机科学系（DCC）巴西矿务联邦大学（UFMG））；School of Computer Science University of Sheffield（计算机科学学院谢菲尔德大学）
