Linguistically Driven Graph Capsule Network for Visual Question Reasoning
Sun Yat-sen University（中山大学）；University of California, Los Angles（加利福尼亚大学洛杉矶分校）
Qingxing Cao, Xiaodan Liang, Keze Wang, Liang Lin
备注：
Submitted to TPAMI 2020. We have achieved an end-to-end interpretable structural reasoning for general images without the requirement of layout annotations
Recently, studies of visual question answering have explored various architectures of end-to-end networks and achieved promising results on both natural and synthetic datasets, which require explicitly compositional reasoning. However, it has been argued that these black-box approaches lack interpretability of results, and thus cannot perform well on generalization tasks due to overfitting the dataset bias. In this work, we aim to combine the benefits of both sides and overcome their limitations to achieve an end-to-end interpretable structural reasoning for general images without the requirement of layout annotations. Inspired by the property of a capsule network that can carve a tree structure inside a regular convolutional neural network (CNN), we propose a hierarchical compositional reasoning model called the "Linguistically driven Graph Capsule Network", where the compositional process is guided by the linguistic parse tree. Specifically, we bind each capsule in the lowest layer to bridge the linguistic embedding of a single word in the original question with visual evidence and then route them to the same capsule if they are siblings in the parse tree. This compositional process is achieved by performing inference on a linguistically driven conditional random field (CRF) and is performed across multiple graph capsule layers, which results in a compositional reasoning process inside a CNN. Experiments on the CLEVR dataset, CLEVR compositional generation test, and FigureQA dataset demonstrate the effectiveness and composition generalization ability of our end-to-end model.
Capsule Lens: Locating and Tracking Concept Geometry in Model Representations
National University of Singapore（新加坡国立大学）；Indian Institute of Technology (ISM) Dhanbad（印度理工学院（ISM）丹巴德分校）；The University of Sydney（悉尼大学）；The Chinese University of Hong Kong（香港中文大学）
University of Sydney（悉尼大学）；University of New South Wales（新南威尔士大学）
Micro-expressions are reflections of people's true feelings and motives, which attract an increasing number of researchers into the study of automatic facial micro-expression recognition. The short detection window, the subtle facial muscle movements, and the limited training samples make micro-expression recognition challenging. To this end, we propose a novel Identity-aware and Capsule-Enhanced Generative Adversarial Network with graph-based reasoning (ICE-GAN), introducing micro-expression synthesis as an auxiliary task to assist recognition. The generator produces synthetic faces with cont…
Cornell University（康奈尔大学）；Allen Institute for AI（艾伦人工智能研究所）；New York University（纽约大学）；Princeton University（普林斯顿大学）
We introduce KiloGram, a resource for studying abstract visual reasoning in humans and machines. Drawing on the history of tangram puzzles as stimuli in cognitive science, we build a richly annotated dataset that, with >1k distinct stimuli, is orders of magnitude larger and more diverse than prior resources. It is both visually and linguistically richer, moving beyond whole shape descriptions to include segmentation maps and part labels. We use this resource to evaluate the abstract visual reasoning capacities of recent multi-modal models. We observe that pre-trained weights demonstrate limite…
Retinal Vessel Segmentation with Deep Graph and Capsule Reasoning
Yangtze Delta Region Institute (Huzhou), University of Electronic Science and Technology of China（电子科技大学长三角研究院（湖州））；School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；University of South Florida（南佛罗里达大学）
Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path Anchoring
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Harbin Institute of Technology, Shenzhen（深圳哈尔滨工业大学）；University of Science and Technology Beijing（北京科技大学）
Cornell University（康奈尔大学）；Cornell Tech（康奈尔科技校区）；University of Maryland（马里兰大学）
We introduce a new dataset for joint reasoning about natural language and images, with a focus on semantic diversity, compositionality, and visual reasoning challenges. The data contains 107,292 examples of English sentences paired with web photographs. The task is to determine whether a natural language caption is true about a pair of photographs. We crowdsource the data using sets of visually rich images and a compare-and-contrast task to elicit linguistically diverse language. Qualitative analysis shows the data requires compositional joint reasoning, including about quantities, comparisons…
AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions
University of Maryland, College Park（马里兰大学帕克分校）；University of Maryland School of Medicine（马里兰大学医学院）；University of Maryland Institute for Health Computing（马里兰大学健康计算研究所）；University of Maryland Medical System（马里兰大学医学系统）
A Graph-Augmented knowledge Distillation based Dual-Stream Vision Transformer with Region-Aware Attention for Gastrointestinal Disease Classification with Explainable AI
Department of Computer Science and Engineering（计算机科学与工程系）
