Human-Object Interaction (HOI) detection devotes to learn how humans interact with surrounding objects via inferring triplets of < human, verb, object >. However, recent HOI detection methods mostly rely on additional annotations (e.g., human pose) and neglect powerful interactive reasoning beyond convolutions. In this paper, we present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs, in which interactive semantics implied among visual targets are efficiently exploited. The proposed model consists of a project function that maps related targets from convolution space to a graph-based semantic space, a message passing process propagating semantics among all nodes and an update function transforming the reasoned nodes back to convolution space. Furthermore, we construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Beyond inferring HOIs using instance features respectively, the framework dynamically parses pairwise interactive semantics among visual targets by integrating two-level in-Graphs, i.e., scene-wide and instance-wide in-Graphs. Our framework is end-to-end trainable and free from costly annotations like human pose. Extensive experiments show that our proposed framework outperforms existing HOI detection methods on both V-COCO and HICO-DET benchmarks and improves the baseline about 9.4% and 15% relatively, validating its efficacy in detecting HOIs.
Human-Object Interaction (HOI) detection devotes to learn how humans interact with surrounding objects. Latest end-to-end HOI detectors are short of relation reasoning, which leads to inability to learn HOI-specific interactive semantics for predictions. In this paper, we therefore propose novel relation reasoning for HOI detection. We first present a progressive Relation-aware Frame, which brings a new structure and parameter sharing pattern for interaction inference. Upon the frame, an Interaction Intensifier Module and a Correlation Parsing Module are carefully designed, where: a) interacti…
Video-CoM: Interactive Video Reasoning via Chain of Manipulations
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；University of California Merced（加州梅尔 Ced 大学）；Google Research（谷歌研究院）；Linköping University（林奈大学）；Australian National University（澳大利亚国立大学）
Human-Object Interaction (HOI) detection plays a core role in activity understanding. Though recent two/one-stage methods have achieved impressive results, as an essential step, discovering interactive human-object pairs remains challenging. Both one/two-stage methods fail to effectively extract interactive pairs instead of generating redundant negative pairs. In this work, we introduce a previously overlooked interactiveness bimodal prior: given an object in an image, after pairing it with the humans, the generated pairs are either mostly non-interactive, or mostly interactive, with the forme…
Human-Object Interaction (HOI) Detection is an important problem to understand how humans interact with objects. In this paper, we explore Interactiveness Knowledge which indicates whether human and object interact with each other or not. We found that interactiveness knowledge can be learned across HOI datasets, regardless of HOI category settings. Our core idea is to exploit an Interactiveness Network to learn the general interactiveness knowledge from multiple HOI datasets and perform Non-Interaction Suppression before HOI classification in inference. On account of the generalization of int…
Manually annotating object segmentation masks is very time consuming. Interactive object segmentation methods offer a more efficient alternative where a human annotator and a machine segmentation model collaborate. In this paper we make several contributions to interactive segmentation: (1) we systematically explore in simulation the design space of deep interactive segmentation models and report new insights and caveats; (2) we execute a large-scale annotation campaign with real human annotators, producing masks for 2.5M instances on the OpenImages dataset. We plan to release this data public…
Towards Open Environments and Instructions: General Vision-Language Navigation via Fast-Slow Interactive Reasoning
School of Artificial Intelligence, College of Intelligence and Computing, Tianjin University, China（人工智能学院，智能与计算学院，天津大学，中国）；School of Computer Science and Information Engineering, Hefei University of Technology, China（计算机科学与信息工程学院，合肥工业大学，中国）
From Perception to Action: An Interactive Benchmark for Vision Reasoning
Singapore University of Technology（新加坡科技设计大学）；Singapore Management University (SMU), Singapore（新加坡管理学院）；Nanyang Technological University (NTU), Singapore（南洋理工大学）；University of Science（科学大学）
Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation
Pengcheng Laboratory（鹏城实验室）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；South China University of Technology（华南理工大学）；Pazhou Lab (Huangpu)（琶洲实验室（黄埔））；UC Merced（加州大学默塞德分校）；Yonsei University（延世大学）
Beijing Jiaotong University（北京交通大学）；University of Technology Sydney（悉尼科技大学）
In this work, we focus on Interactive Human Parsing (IHP), which aims to segment a human image into multiple human body parts with guidance from users' interactions. This new task inherits the class-aware property of human parsing, which cannot be well solved by traditional interactive image segmentation approaches that are generally class-agnostic. To tackle this new task, we first exploit user clicks to identify different human parts in the given image. These clicks are subsequently transformed into semantic-aware localization maps, which are concatenated with the RGB image to form the input…
TextIM: Part-aware Interactive Motion Synthesis from Text
School of Computer Science, Wuhan University（计算机学院,武汉大学）；Institute of Artificial Intelligence, Wuhan University（人工智能研究院,武汉大学）；Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（湖北省多媒体与网络通信工程重点实验室,武汉大学）
HUMOF: Human Motion Forecasting in Interactive Social Scenes
ShanghaiTech University（上海科技大学）；The University of Hong Kong（香港大学）；Digital Trust Centre, Nanyang Technological University（南洋理工大学数字信任中心）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）
Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Tasks
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Alibaba Group（阿里巴巴集团）；DAMO Academy, Alibaba Group（阿里巴巴集团大模型学院）；Nanjing Institute of Software Technology（南京软件技术研究所）；Nanjing University of Posts and Telecommunications（南京邮电大学）；Hohai University（河海大学）
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；National Key Laboratory of General Artificial Intelligence, BIGAI（北京智源人工智能研究院通用人工智能国家重点实验室）；Department of Automation, Tsinghua University（清华大学自动化系）
Georgia Institute of Technology（佐治亚理工学院）；Intel（英特尔）
Skeleton-based human action recognition technologies are increasingly used in video based applications, such as home robotics, healthcare on aging population, and surveillance. However, such models are vulnerable to adversarial attacks, raising serious concerns for their use in safety-critical applications. To develop an effective defense against attacks, it is essential to understand how such attacks mislead the pose detection models into making incorrect predictions. We present SkeletonVis, the first interactive system that visualizes how the attacks work on the models to enhance human under…
Shanghaitech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
Generating ``bullet-time'' effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design ability for human free-viewpoint rendering, especially under the real-time, dynamic and general setting for our trajectory-aware task. To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effe…
Compressing Human Body Video with Interactive Semantics: A Generative Approach
City University of Hong Kong（香港城市大学）；Alibaba DAMO Academy & Hupan Laboratory（阿里达摩院及胡畔实验室）；Shenzhen Research Institute, City University of Hong Kong（香港城市大学深圳研究院）
Beijing Institute of Technology（北京理工大学）；Peking University（北京大学）；Yangtze Delta Region Academy of Beijing Institute of Technology（北京理工大学长三角研究院）；BIGAI（北京通用人工智能研究院）
Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）；Shanghai Qi Zhi Institute（上海期智研究院）
Human-Object Interaction (HOI) detection is an important problem to understand how humans interact with objects. In this paper, we explore interactiveness knowledge which indicates whether a human and an object interact with each other or not. We found that interactiveness knowledge can be learned across HOI datasets and bridge the gap between diverse HOI category settings. Our core idea is to exploit an interactiveness network to learn the general interactiveness knowledge from multiple HOI datasets and perform Non-Interaction Suppression (NIS) before HOI classification in inference. On accou…
MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；UMass Amherst（马萨诸塞大学阿默斯特分校）；Tsinghua University（清华大学）
Large pre-trained vision and language models have demonstrated remarkable capacities for various tasks. However, solving the knowledge-based visual reasoning tasks remains challenging, which requires a model to comprehensively understand image content, connect the external world knowledge, and perform step-by-step reasoning to answer the questions correctly. To this end, we propose a novel framework named Interactive Prompting Visual Reasoner (IPVR) for few-shot knowledge-based visual reasoning. IPVR contains three stages, see, think and confirm. The see stage scans the image and grounds the v…
CoReLIN: Constraint-based Reasoning for Zero-shot Lifelong Interactive Navigation
Purdue University（普渡大学）；Indian Institute of Technology, Kharagpur（印度理工学院卡哈拉格普尔分校）；DEVCOM Army Research Lab（DEVCOM陆军研究实验室）；National University of Singapore（新加坡国立大学）
Guide, Think, Act: Interactive Embodied Reasoning in Vision-Language-Action Models
Futian Laboratory（福田实验室）；Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；School of Robotics, Hunan University（湖南大学机器人学院）；South China University of Technology（华南理工大学）；Visincept（Visincept公司）；National Key Laboratory of Smart Farm Technologies and Systems（智能农业技术与系统国家重点实验室）
Boston University（波士顿大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）
In recent years, vision-language research has shifted to study tasks which require more complex reasoning, such as interactive question answering, visual common sense reasoning, and question-answer plausibility prediction. However, the datasets used for these problems fail to capture the complexity of real inputs and multimodal environments, such as ambiguous natural language requests and diverse digital domains. We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a dataset with natural language commands for the greatest number of interactive environments to date. MoTIF is the first…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the…
This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The core issue here is the grounding of referring expressions: infer objects and their relationships from input images and language expressions. INGRESS allows for unconstrained object categories and unconstrained language expressions. Further, it asks questions to disambiguate referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural network model for grounding. The first stage uses a neural…
