GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science, The University of Sydney（悉尼大学计算机科学学院）；Department of Computer Science and Technology, Nanjing University（南京大学计算机科学与技术系）；Artificial Intelligence Industrial Technology Research Institute, Nanjing Institute of Technology（南京工程学院人工智能产业技术研究院）
Vision-Language Models (VLMs), pre-trained on large-scale datasets, have shown impressive performance in various visual recognition tasks. This advancement paves the way for notable performance in Zero-Shot Egocentric Action Recognition (ZS-EAR). Typically, VLMs handle ZS-EAR as a global video-text matching task, which often leads to suboptimal alignment of vision and linguistic knowledge. We propose a refined approach for ZS-EAR using VLMs, emphasizing fine-grained concept-description alignment that capitalizes on the rich semantic and contextual details in egocentric videos. In this paper, we introduce GPT4Ego, a straightforward yet remarkably potent VLM framework for ZS-EAR, designed to enhance the fine-grained alignment of concept and description between vision and language. Extensive experiments demonstrate GPT4Ego significantly outperforms existing VLMs on three large-scale egocentric video benchmarks, i.e., EPIC-KITCHENS-100 (33.2%, +9.4%), EGTEA (39.6%, +5.5%), and CharadesEgo (31.5%, +2.6%).
GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition
School of Cyber Science and Engineering, Southeast University（东南大学网络空间安全学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Purple Mountain Laboratories（紫金山实验室）
University of Colorado, Boulder（科罗拉多大学博尔德分校）；Facebook Reality Labs（Facebook现实实验室）
Although there has been significant research in egocentric action recognition, most methods and tasks, including EPIC-KITCHENS, suppose a fixed set of action classes. Fixed-set classification is useful for benchmarking methods, but is often unrealistic in practical settings due to the compositionality of actions, resulting in a functionally infinite-cardinality label set. In this work, we explore generalization with an open set of classes by unifying two popular approaches: few- and zero-shot generalization (the latter which we reframe as cross-modal few-shot generalization). We propose a new …
Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
Department of Computer Science and Engineering, University at Buffalo（计算机科学与工程系，布法罗大学）；Department of Surgery, University at Buffalo（外科系，布法罗大学）；Department of Computer Science and Engineering, IIT, Jodhpur（计算机科学与工程系，印度理工学院，乔杜尔）；Department of Computer Science and Engineering, University of Notre Dame（计算机科学与工程系，圣母大学）
Unleashing Spatial Reasoning in Multimodal Large Language Models via Textual Representation Guided Reasoning
College of AI, Tsinghua University, Beijing, China（清华大学人工智能学院，北京，中国）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室，上海，中国）；The University of Tokyo, Tokyo, Japan（东京大学，东京，日本）
Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation
GenAI, Meta（Meta GenAI部门）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Sungkyunkwan University（成均馆大学）；Georgia Institute of Technology（佐治亚理工学院）
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；King Abdullah University of Science and Technology（科廷大学）；Fudan University（复旦大学）
We benchmark contemporary action recognition models (TSN, TRN, and TSM) on the recently introduced EPIC-Kitchens dataset and release pretrained models on GitHub (https://github.com/epic-kitchens/action-models) for others to build upon. In contrast to popular action recognition datasets like Kinetics, Something-Something, UCF101, and HMDB51, EPIC-Kitchens is shot from an egocentric perspective and captures daily actions in-situ. In this report, we aim to understand how well these models can tackle the challenges present in this dataset, such as its long tail class distribution, unseen environme…
MSGNav: Unleashing the Power of Multi-modal 3D Scene Graph for Zero-Shot Embodied Navigation
Fujian Key Laboratory of Urban Intelligent Sensing and Computing（福建智能感知与计算重点实验室）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China（多媒体可信感知与高效计算重点实验室）；Beihang University（北航）；Nanyang Technological University（南洋理工大学）；University of Chinese Academy of Sciences（中国科学院大学）；Zhongguancun Academy（中关村学院）
In this abstract we describe recent [4,7] and latest work on the determination of affordances in visually perceived 3D scenes. Our method builds on the hypothesis that geometry on its own provides enough information to enable the detection of significant interaction possibilities in the environment. The motivation behind this is that geometric information is intimately related to the physical interactions afforded by objects in the world. The approach uses a generic representation for the interaction between everyday objects such as a mug or an umbrella with the environment, and also for more …
Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts
City University of Hong Kong（香港城市大学）；Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of Oxford（牛津大学）；Harvard University（哈佛大学）；Nanyang Technological University（南洋理工大学）
We present an approach for identifying picturesque highlights from large amounts of egocentric video data. Given a set of egocentric videos captured over the course of a vacation, our method analyzes the videos and looks for images that have good picturesque and artistic properties. We introduce novel techniques to automatically determine aesthetic features such as composition, symmetry and color vibrancy in egocentric videos and rank the video frames based on their photographic qualities to generate highlights. Our approach also uses contextual information such as GPS, when available, to asse…
University of Barcelona（巴塞罗那大学）；Institut de Robòtica i Informàtica Industrial (CSIC-UPC)（工业机器人信息学研究所（CSIC-UPC））
In this paper, we propose a new approach to under-stand actions in egocentric videos that exploits the semantics of object interactions at both frame and temporal levels. At the frame level, we use a region-based approach that takes as input a primary region roughly corresponding to the user hands and a set of secondary regions potentially corresponding to the interacting objects and calculates the action score through a CNN formulation. This information is then fed to a Hierarchical LongShort-Term Memory Network (HLSTM) that captures temporal dependencies between actions within and across sho…
Egocentric cameras are being worn by an increasing number of users, among them many security forces worldwide. GoPro cameras already penetrated the mass market, reporting substantial increase in sales every year. As head-worn cameras do not capture the photographer, it may seem that the anonymity of the photographer is preserved even when the video is publicly distributed. We show that camera motion, as can be computed from the egocentric video, provides unique identity information. The photographer can be reliably recognized from a few seconds of video captured when walking. The proposed meth…
Inspired by the remarkable ability of the infant visual learning system, a recent study collected first-person images from children to analyze the `training data' that they receive. We conduct a follow-up study that investigates two additional directions. First, given that infants can quickly learn to recognize a new object without much supervision (i.e. few-shot learning), we limit the number of training images. Second, we investigate how children control the supervision signals they receive during learning based on hand manipulation of objects. Our experimental results suggest that supervisi…
Mass utilization of body-worn cameras has led to a huge corpus of available egocentric video. Existing video summarization algorithms can accelerate browsing such videos by selecting (visually) interesting shots from them. Nonetheless, since the system user still has to watch the summary videos, browsing large video databases remain a challenge. Hence, in this work, we propose to generate a textual synopsis, consisting of a few sentences describing the most important events in a long egocentric videos. Users can read the short text to gain insight about the video, and more importantly, efficie…
Four Eyes Are Better Than Two: Harnessing the Collaborative Potential of Large Models via Differentiated Thinking and Complementary Ensembles
Lenovo Research（联想研究院）；Tsinghua University（清华大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences(CAS)（中国科学院自动化研究所）；Zhongguancun Academy（中关村学院）；University of Chinese Academy of Sciences（中国科学院大学）
How You Move Tells What You'll Do: Trajectory-Conditioned Egocentric Prediction
Khoury College of Computer Sciences, Northeastern University, Boston（北德文斯克学院，东北大学，波士顿）；Korea Advanced Institute of Science and Technology, Daejeon（韩国科学技术院，大田）；Department of Mathematics and Computer Science, University of Catania, Italy（卡塔尼亚大学数学与计算机科学系，意大利）
Vector Institute for AI（向量人工智能研究所）；Meta AI；University of British Columbia（不列颠哥伦比亚大学）
Video understanding tasks take many forms, from action detection to visual query localization and spatio-temporal grounding of sentences. These tasks differ in the type of inputs (only video, or video-query pair where query is an image region or sentence) and outputs (temporal segments or spatio-temporal tubes). However, at their core they require the same fundamental understanding of the video, i.e., the actors and objects in it, their actions and interactions. So far these tasks have been tackled in isolation with individual, highly specialized architectures, which do not exploit the interpl…
One-Shot Affordance Grounding of Deformable Objects in Egocentric Organizing Scenes
School of Artificial Intelligence and Robotics, Hunan University, China（人工智能与机器人学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
