Operating rooms (ORs) demand precise coordination among surgeons, nurses, and equipment in a fast-paced, occlusion-heavy environment, necessitating advanced perception models to enhance safety and efficiency. Existing datasets either provide partial egocentric views or sparse exocentric multi-view context, but do not explore the comprehensive combination of both. We introduce EgoExOR, the first OR dataset and accompanying benchmark to fuse first-person and third-person perspectives. Spanning 94 minutes (84,553 frames at 15 FPS) of two emulated spine procedures, Ultrasound-Guided Needle Insertion and Minimally Invasive Spine Surgery, EgoExOR integrates egocentric data (RGB, gaze, hand tracking, audio) from wearable glasses, exocentric RGB and depth from RGB-D cameras, and ultrasound imagery. Its detailed scene graph annotations, covering 36 entities and 22 relations (568,235 triplets), enable robust modeling of clinical interactions, supporting tasks like action recognition and human-centric perception. We evaluate the surgical scene graph generation performance of two adapted state-of-the-art models and offer a new baseline that explicitly leverages EgoExOR's multimodal and multi-perspective signals. This new dataset and benchmark set a new foundation for OR perception, offering a rich, multimodal resource for next-generation clinical perception.
Facebook AI Research（Facebook AI研究院）；UT Austin（德克萨斯大学奥斯汀分校）
We introduce an approach for pre-training egocentric video models using large-scale third-person video datasets. Learning from purely egocentric data is limited by low dataset scale and diversity, while using purely exocentric (third-person) data introduces a large domain mismatch. Our idea is to discover latent signals in third-person video that are predictive of key egocentric-specific properties. Incorporating these signals as knowledge distillation losses during pre-training results in models that benefit from both the scale and diversity of third-person video data, as well as representati…
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
OpenGVLab, Shanghai AI Laboratory（上海人工智能实验室OpenGVLab）；The University of Tokyo（东京大学）；Nanjing University（南京大学）；Fudan University（复旦大学）；Zhejiang University（浙江大学）；USTC（中国科学技术大学）；SIAT, CAS（中国科学院深圳先进技术研究院）
First-person (egocentric) and third person (exocentric) videos are drastically different in nature. The relationship between these two views have been studied in recent years, however, it has yet to be fully explored. In this work, we introduce two datasets (synthetic and natural/real) containing simultaneously recorded egocentric and exocentric videos. We also explore relating the two domains (egocentric and exocentric) in two aspects. First, we synthesize images in the egocentric domain from the exocentric domain using a conditional generative adversarial network (cGAN). We show that with en…
University of Science and Technology of China（中国科学技术大学）；The University of Sydney（悉尼大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
JD Explore Academy（京东探索研究院）；University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；The University of Sydney（悉尼大学）
Affordance grounding, a task to ground (i.e., localize) action possibility region in objects, which faces the challenge of establishing an explicit link with object parts due to the diversity of interactive affordance. Human has the ability that transform the various exocentric interactions to invariant egocentric affordance so as to counter the impact of interactive diversity. To empower an agent with such ability, this paper proposes a task of affordance grounding from exocentric view, i.e., given exocentric human-object interaction and egocentric object images, learning the affordance knowl…
EgoPrivacy: What Your First-Person Camera Says About You?
University of California, San Diego（加州大学圣地亚哥分校）；Qualcomm AI Research, an initiative of Qualcomm Technologies, Inc（高通人工智能研究）；University of Electronic Science and Technology of China（电子科技大学）
Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学AI研究院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）；MoE Key Lab of AI, School of Computer Science, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学计算机学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Lenovo（联想公司）
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Fraunhofer IPK（弗劳恩霍夫研究所）；Technical University of Berlin（柏林技术大学）；University of Tübingen（图宾根大学）；RWTH Aachen University（亚琛工业大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos
Fudan University（复旦大学）；The University of Tokyo（东京大学）；Zhejiang University（浙江大学）；Hong Kong University of Science and Technology（香港科技大学）；Nanjing University（南京大学）；Shanghai Jiao Tong University（上海交通大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；University of Tokyo（东京大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）
EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）；School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）
Texas State University（德克萨斯州立大学）；University of Trento（特伦托大学）；Cisco（思科）
Cross-view image generation has been recently proposed to generate images of one view from another dramatically different view. In this paper, we investigate exocentric (third-person) view to egocentric (first-person) view image generation. This is a challenging task since egocentric view sometimes is remarkably different from exocentric view. Thus, transforming the appearances across the two views is a non-trivial task. To this end, we propose a novel Parallel Generative Adversarial Network (P-GAN) with a novel cross-cycle loss to learn the shared information for generating egocentric images …
Robust Ego-Exo Correspondence with Long-Term Memory
University of Chinese Academy of Sciences（中国科学院大学）；University of North Texas（北德克萨斯大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；Rochester Institute of Technology（罗切斯特理工学院）
Mirror neurons have been observed in the primary motor cortex of primate species, in particular in humans and monkeys. A mirror neuron fires when a person performs a certain action, and also when he observes the same action being performed by another person. A crucial step towards building fully autonomous intelligent systems with human-like learning abilities is the capability in modeling the mirror neuron. On one hand, the abundance of egocentric cameras in the past few years has offered the opportunity to study a lot of vision problems from the first-person perspective. A great deal of inte…
Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos
The University of Tokyo（东京大学）；OMRON SINIC X Corp.（欧姆龙SINIC X公司）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）；LY Corporation（LY公司）
