To understand a complex action, multiple sources of information, including appearance, positional, and semantic features, need to be integrated. However, these features are difficult to be fused since they often differ significantly in modality and dimensionality. In this paper, we present a novel framework that accomplishes this goal by interactive fusion, namely, projecting features across different spaces and guiding it using an auxiliary prediction task. Specifically, we implement the framework in three steps, namely, positional-to-appearance feature extraction, semantic feature interaction, and semantic-to-positional prediction. We evaluate our approach on two action recognition datasets, Something-Something and Charades. Interactive fusion achieves consistent accuracy gain beyond off-the-shelf action recognition algorithms. In particular, on Something-Else, the compositional setting of Something-Something, interactive fusion reports a remarkable gain of 2.9% in terms of top-1 accuracy.
We present a general framework for compositional action recognition -- i.e. action recognition where the labels are composed out of simpler components such as subjects, atomic-actions and objects. The main challenge in compositional action recognition is that there is a combinatorially large set of possible actions that can be composed using basic components. However, compositionality also provides a structure that can be exploited. To do so, we develop and test a novel Structured Attention Fusion (SAF) self-attention mechanism to combine information from object detections, which capture the t…
Recognizing human actions is fundamentally a spatio-temporal reasoning problem, and should be, at least to some extent, invariant to the appearance of the human and the objects involved. Motivated by this hypothesis, in this work, we take an object-centric approach to action recognition. Multiple works have studied this setting before, yet it remains unclear (i) how well a carefully crafted, spatio-temporal layout-based method can recognize human actions, and (ii) how, and when, to fuse the information from layout and appearance-based models. The main focus of this paper is compositional/few-s…
Shanghai Jiao Tong University（上海交通大学）；The Second Academy of China Aerospace Science And Industry Corporation（中国航天科工集团第二研究院）
Multiple object tracking (MOT) is a significant task in achieving autonomous driving. Traditional works attempt to complete this task, either based on point clouds (PC) collected by LiDAR, or based on images captured from cameras. However, relying on one single sensor is not robust enough, because it might fail during the tracking process. On the other hand, feature fusion from multiple modalities contributes to the improvement of accuracy. As a result, new techniques based on different sensors integrating features from multiple modalities are being developed. Texture information from RGB came…
National Technical University of Athens（雅典国立技术大学）；Institute for Speech and Language Processing（言语与语言处理研究所）
We propose a novel deep architecture for the task of reasoning about social interactions in videos. We leverage the multi-step reasoning capabilities of Compositional Attention Networks (MAC), and propose a multimodal extension (MAC-X). MAC-X is based on a recurrent cell that performs iterative mid-level fusion of input modalities (visual, auditory, text) over multiple reasoning steps, by use of a temporal attention mechanism. We then combine MAC-X with LSTMs for temporal input processing in an end-to-end architecture. Our ablation studies show that the proposed MAC-X architecture can effectiv…
Inception Institute of Artificial Intelligence（人工智能研究院）；Tianjin University（天津大学）；University of California, Los Angeles（加州大学洛杉矶分校）
This work proposes to combine neural networks with the compositional hierarchy of human bodies for efficient and complete human parsing. We formulate the approach as a neural information fusion framework. Our model assembles the information from three inference processes over the hierarchy: direct inference (directly predicting each part of a human body using image information), bottom-up inference (assembling knowledge from constituent parts), and top-down inference (leveraging context from parent nodes). The bottom-up and top-down inferences explicitly model the compositional and decompositi…
Liaoning Normal University（辽宁师范大学）；Dalian University of Technology（大连理工大学）
General deep learning-based methods for infrared and visible image fusion rely on the unsupervised mechanism for vital information retention by utilizing elaborately designed loss functions. However, the unsupervised mechanism depends on a well designed loss function, which cannot guarantee that all vital information of source images is sufficiently extracted. In this work, we propose a novel interactive feature embedding in self-supervised learning framework for infrared and visible image fusion, attempting to overcome the issue of vital information degradation. With the help of self-supervis…
In this paper, we propose a novel fully convolutional two-stream fusion network (FCTSFN) for interactive image segmentation. The proposed network includes two sub-networks: a two-stream late fusion network (TSLFN) that predicts the foreground at a reduced resolution, and a multi-scale refining network (MSRN) that refines the foreground at full resolution. The TSLFN includes two distinct deep streams followed by a fusion network. The intuition is that, since user interactions are more direct information on foreground/background than the image itself, the two-stream structure of the TSLFN reduce…
Inclusive Interactive Collisions for Multi-View Consistent Compositional 3D Generation
School of Computer Science and Technology, China University of Petroleum (East China)（中国石油大学（华东）计算机科学与技术学院）；Artificial Intelligence Research Institute, Shenzhen University of Advanced Technology（深圳理工大学人工智能研究院）；College of Computer Science, Northwest University（西北大学计算机科学学院）
Weave Forcing: Compositional Memory Routing for Interactive Long Video Generation
University of Electronic Science and Technology of China (UESTC)（电子科技大学）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）
University of Bonn（波恩大学）；Indian Institute of Technology Bombay（印度理工学院孟买分校）；National University of Singapore（新加坡国立大学）
Segmenting objects of interest in an image is an essential building block of applications such as photo-editing and image analysis. Under interactive settings, one should achieve good segmentations while minimizing user input. Current deep learning-based interactive segmentation approaches use early fusion and incorporate user cues at the image input layer. Since segmentation CNNs have many layers, early fusion may weaken the influence of user interactions on the final prediction results. As such, we propose a new multi-stage guidance framework for interactive segmentation. By incorporating us…
Northwestern Polytechnical University（西北工业大学）；The Hong Kong Polytechnic University（香港理工大学）
Compositional Zero-Shot Learning (CZSL) aims to recognize novel concepts formed by known states and objects during training. Existing methods either learn the combined state-object representation, challenging the generalization of unseen compositions, or design two classifiers to identify state and object separately from image features, ignoring the intrinsic relationship between them. To jointly eliminate the above issues and construct a more robust CZSL system, we propose a novel framework termed Decomposed Fusion with Soft Prompt (DFSP)1, by involving vision-language models (VLMs) for unsee…
Progress in image captioning is gradually getting complex as researchers try to generalized the model and define the representation between visual features and natural language processing. This work tried to define such kind of relationship in the form of representation called Tensor Product Representation (TPR) which generalized the scheme of language modeling and structuring the linguistic attributes (related to grammar and parts of speech of language) which will provide a much better structure and grammatically correct sentence. TPR enables better and unique representation and structuring o…
We present Modular interactive VOS (MiVOS) framework which decouples interaction-to-mask and mask propagation, allowing for higher generalizability and better performance. Trained separately, the interaction module converts user interactions to an object mask, which is then temporally propagated by our propagation module using a novel top-$k$ filtering strategy in reading the space-time memory. To effectively take the user's intent into account, a novel difference-aware module is proposed to learn how to properly fuse the masks before and after each interaction, which are aligned with the targ…
LightFuse: Relightable Interactive Gaussian Scene Reconstruction via Multi-Scan Fusion and 2D Gaussian Ray Tracing
ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；College of Mathematics, Sichuan University（四川大学数学学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Chengdu Minto Tech（成都敏途科技）
Taiyuan University of Science and Technology（太原科技大学）；Wenzhou University（温州大学）
The existing generative adversarial fusion methods generally concatenate source images and extract local features through convolution operation, without considering their global characteristics, which tends to produce an unbalanced result and is biased towards the infrared image or visible image. Toward this end, we propose a novel end-to-end mode based on generative adversarial training to achieve better fusion balance, termed as \textit{interactive compensatory attention fusion network} (ICAFusion). In particular, in the generator, we construct a multi-level encoder-decoder network with a tr…
Towards Accurate Camouflaged Object Detection with Mixture Convolution and Interactive Fusion
National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University（西北工业大学计算机学院）；Center for Brain Imaging Science and Technology, Zhejiang University（浙江大学）；School of Computer Science, China University of Geosciences（中国地质大学计算机学院）；Optical Electrical and Computer Engineering, University of Shanghai for Science and Technology（上海理工大学）；School of Electrical Engineering & Information Department, Northeast Petroleum University（东北石油大学）；School of Electronic, Electrical Engineering and Physics, Fujian University of Technology（福建工程学院）
Interactive Spatial-Frequency Fusion Mamba for Multi-Modal Image Fusion
School of Future Technology, Dalian University of Technology and the Key Laboratory of Data Science and Smart Education (Hainan Normal University), Ministry of Education（未来技术学院，大连理工大学和数据科学与智能教育关键实验室（海南师范大学），教育部）；Affiliated Zhongshan Hospital of Dalian University（大连大学附属中山医院）；School of Computer Science and Artificial Intelligence, Wuhan University of Technology（计算机科学与人工智能学院，武汉理工大学）；Central Hospital of Dalian University of Technology（大连理工大学中心医院）；School of Information and Communication Engineering, Dalian University of Technology（信息与通信工程学院，大连理工大学）
Lightweight Method for Interactive 3D Medical Image Segmentation with Multi-Round Result Fusion
Aerospace Center Hospital（航天中心医院）；Yizhun Medical AI Co., Ltd（医准智能医疗科技有限公司）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Southern University of Science and Technology（南方科技大学）；Beijing Institute of Technology（北京理工大学）
Northwestern Polytechnical University（西北工业大学）；Nanjing University（南京大学）；National University of Defense Technology（国防科技大学）
As a concrete application of multi-view learning, multi-view classification improves the traditional classification methods significantly by integrating various views optimally. Although most of the previous efforts have been demonstrated the superiority of multi-view learning, it can be further improved by comprehensively embedding more powerful cross-view interactive information and a more reliable multi-view fusion strategy in intensive studies. To fulfill this goal, we propose a novel multi-view learning framework to make the multi-view classification better aimed at the above-mentioned tw…
RefGlass-GS: A UAV-Enabled Fusion Framework for Photorealistic, Semantic and Interactive Digitization of Reflective Glass Facades via Gaussian Splatting
Department of Civil and Environmental Engineering, The Hong Kong University of Science and Technology（香港科技大学土木与环境工程系）；Department of Civil and Environmental Engineering, University of Maryland（马里兰大学土木与环境工程系）；Department of the Built Environment, National University of Singapore（新加坡国立大学环境与建筑系）；Department of Architecture and Civil Engineering, City University of Hong Kong（香港城市大学建筑与土木工程系）
ASSIST: Interactive Scene Nodes for Scalable and Realistic Indoor Simulation
Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院（AIR））；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Beijing University of Civil Engineering and Architecture（北京建筑大学）；University of Science and Technology of China（中国科学技术大学）；McGill University（麦吉尔大学）；Beijing Institute（北京研究所）
I2E: From Image Pixels to Actionable Interactive Environments for Text-Guided Image Editing
Huazhong University of Science and Technology（华中科技大学）；Kuaishou Technology（快手科技）；Central China Normal University（华中师范大学）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）
Nankai University（南开大学）；Tianjin University of Finance and Economics（天津财经大学）；Tsinghua University（清华大学）
Salient object detection (SOD) focuses on distinguishing the most conspicuous objects in the scene. However, most related works are based on RGB images, which lose massive useful information. Accordingly, with the maturity of thermal technology, RGB-T (RGB-Thermal) multi-modality tasks attain more and more attention. Thermal infrared images carry important information which can be used to improve the accuracy of SOD prediction. To accomplish it, the methods to integrate multi-modal information and suppress noises are critical. In this paper, we propose a novel network called Interactive Contex…
Multi-view Graph Convolutional Network with Fully Leveraging Consistency via Granular-ball-based Topology Construction, Feature Enhancement and Interactive Fusion
School of Computer, Jiangsu University of Science（江苏科技大学计算机学院）；Chongqing Key Laboratory of Computational Intelligence, Chongqing University of Posts（重庆邮电大学计算智能重点实验室）；College of Computer（计算机学院）；Data Science, Fuzhou University, Fuzhou 350116, China（福州大学数据科学学院）
A Spatial-Spectral-Frequency Interactive Network for Multimodal Remote Sensing Classification
Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）；School of Information and Electronics, Beijing Institute of Technology（北京理工大学信息与电子学院）；Beijing Key Laboratory of Fractional Signals and Systems, Beijing Institute of Technology（北京理工大学分数域信号与系统北京市重点实验室）；School of Electronic Engineering, Xidian University（西安电子科技大学电子工程学院）；Key Laboratory of Collaborative Intelligent Systems of Ministry of Education, Xidian University（西安电子科技大学教育部协同智能系统重点实验室）；Academy of Artificial Intelligence, Inner Mongolia Normal University（内蒙古师范大学人工智能研究院）
We present a dual-pathway approach for recognizing fine-grained interactions from videos. We build on the success of prior dual-stream approaches, but make a distinction between the static and dynamic representations of objects and their interactions explicit by introducing separate motion and object detection pathways. Then, using our new Motion-Guided Attention Fusion module, we fuse the bottom-up features in the motion pathway with features captured from object detections to learn the temporal aspects of an action. We show that our approach can generalize across appearance effectively and r…
Annotating tens or hundreds of tiny objects in a given image is laborious yet crucial for a multitude of Computer Vision tasks. Such imagery typically contains objects from various categories, yet the multi-class interactive annotation setting for the detection task has thus far been unexplored. To address these needs, we propose a novel interactive annotation method for multiple instances of tiny objects from multiple classes, based on a few point-based user inputs. Our approach, C3Det, relates the full image context with annotator inputs in a local and global manner via late-fusion and featu…
Single-branch object detection methods use shared features for localization and classification, yet the shared features are not fit for the two different tasks simultaneously. Multi-branch object detection methods usually use different features for localization and classification separately, ignoring the relevance between different tasks. Therefore, we propose multi-semantic interactive learning (MSIL) to mine the semantic relevance between different branches and extract multi-semantic enhanced features of objects. MSIL first performs semantic alignment of regression and classification branche…
Hanyang University（汉阳大学）；Korea Advanced Institute of Science & Technology (KAIST)（韩国科学技术院（KAIST））
Fusing data from cameras and LiDAR sensors is an essential technique to achieve robust 3D object detection. One key challenge in camera-LiDAR fusion involves mitigating the large domain gap between the two sensors in terms of coordinates and data distribution when fusing their features. In this paper, we propose a novel camera-LiDAR fusion architecture called, 3D Dual-Fusion, which is designed to mitigate the gap between the feature representations of camera and LiDAR data. The proposed method fuses the features of the camera-view and 3D voxel-view domain and models their interactions through …
EvRWKV: A Continuous Interactive RWKV Framework for Effective Event-Guided Low-Light Image Enhancement
Anhui Provincial International Joint Research center for Advanced technology in Medical imaging（安徽省国际联合先进医学影像技术研究中心）；School of Artificial Intelligence（人工智能学院）；Anhui University（安徽大学）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Secure Artificial Intelligence（安徽省安全人工智能重点实验室）；Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（粤港澳大湾区人工智能与数字经济实验室（深圳））
MMDrive: Interactive Scene Understanding Beyond Vision with Multi-representational Fusion
organization= College of Computer Science；Technology, Jilin University , city= Changchun , country= China；organization= Georgia Institute of Technology , city= Atlanta , country= USA；organization= Qingdao Institute of Software, College of Computer Science；Technology, China University of Petroleum (East China) , city= Qingdao , country= China；organization= Institute for Math \& AI, Wuhan University , city= Wuhan , country= China；organization= University of Michigan, Ann Arbor , country= USA；organization= Thrust of Artificial Intelligence, Hong Kong University of Science；organization= School of Artificial Intelligence；Computer Science, Nantong University , city= Nantong , country= China
RGB-D semantic segmentation methods conventionally use two independent encoders to extract features from the RGB and depth data. However, there lacks an effective fusion mechanism to bridge the encoders, for the purpose of fully exploiting the complementary information from multiple modalities. This paper proposes a novel bottom-up interactive fusion structure to model the interdependencies between the encoders. The structure introduces an interaction stream to interconnect the encoders. The interaction stream not only progressively aggregates modality-specific features from the encoders but a…
C2P-VAR: Continual and Compositional Personalization in Visual Autoregressive Models
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）；South China University of Technology（华南理工大学）
