Gaze object prediction is a newly proposed task that aims to discover the objects being stared at by humans. It is of great application significance but still lacks a unified solution framework. An intuitive solution is to incorporate an object detection branch into an existing gaze prediction method. However, previous gaze prediction methods usually use two different networks to extract features from scene image and head image, which would lead to heavy network architecture and prevent each branch from joint optimization. In this paper, we build a novel framework named GaTector to tackle the gaze object prediction problem in a unified way. Particularly, a specific-general-specific (SGS) feature extractor is firstly proposed to utilize a shared backbone to extract general features for both scene and head images. To better consider the specificity of inputs and tasks, SGS introduces two input-specific blocks before the shared backbone and three task-specific blocks after the shared backbone. Specifically, a novel Defocus layer is designed to generate object-specific features for the object detection task without losing information or requiring extra computations. Moreover, the energy aggregation loss is introduced to guide the gaze heatmap to concentrate on the stared box. In the end, we propose a novel wUoC metric that can reveal the difference between boxes even when they share no overlapping area. Extensive experiments on the GOO dataset verify the superiority of our method in all three tracks, i.e. object detection, gaze estimation, and gaze object prediction.
Xi’an University of Architecture and Technology(西安建筑科技大学);Beijing Institute of Technology(北京理工大学);University of Science and Technology of China(中国科学技术大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Boosting Gaze Object Prediction via Pixel-level Supervision from Vision Foundation Model
Xi’an University of Architecture and Technology(西安建筑科技大学);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院)
Personal Fixations-Based Object Segmentation with Object Localization and Boundary Preservation
Shanghai Institute for Advanced Communication and Data Science(上海先进通信与数据科学研究院);Shanghai University(上海大学);School of Communication and Information Engineering, Shanghai University(上海大学通信与信息工程学院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);Stony Brook University(石溪大学)
GLANCE: Gaze-Led Attention Network for Compressed Edge-inference
Department of Electrical and Computer Engineering, University of Illinois Chicago(伊利诺伊大学芝加哥分校电子与计算机工程系);Department of Electrical and Computer Engineering, New Jersey Institute of Technology(新泽西理工学院电子与计算机工程系);Department of Electrical and Computer Engineering, The University of Texas at Austin(德克萨斯大学奥斯汀分校电子与计算机工程系)
Multi-scale Object-Aware Gaze Estimation via Geometric Reasoning
Qingdao Institute of Software & School of Computer Science and Technology, China University of Petroleum (East China), China Shandong Key Laboratory of Intelligent Oil & Gas Industrial Software, China(青岛软件研究所及中国石油大学(华东)计算机科学与技术学院,山东智能油气工业软件重点实验室,中国)
HOIGaze: Gaze Estimation During Hand-Object Interactions in Extended Reality Exploiting Eye-Hand-Head Coordination
University of Stuttgart Germany(斯图加特大学);University of Tuebingen Germany(图宾根大学);The Center for Bionic Intelligence Tuebingen Stuttgart Germany(图宾根-斯图加特生物智能中心)
From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey(萨里大学视觉、语音与信号处理中心);Surrey Institute for People-Centred AI, University of Surrey(萨里大学以人为本人工智能研究所)
Gaze Embeddings for Zero-Shot Image Classification
Max Planck Institute for Informatics(马克斯·普朗克信息学研究所);Amsterdam Machine Learning Lab(阿姆斯特丹机器学习实验室);Saarland Informatics Campus(萨尔兰信息学园区);University of Amsterdam(阿姆斯特丹大学);Eyeem
Human-Object Interaction Prediction in Videos through Gaze Following
Technical University of Munich (TUM)(慕尼黑工业大学);Technische Universität Wien (TU Wien)(维也纳工业大学);Ulsan National Institute of Science and Technology (UNIST)(蔚山国立科学技术院);German Aerospace Center (DLR)(德国航空航天中心)
Toddlers' Active Gaze Behavior Supports Self-Supervised Object Learning
Frankfurt Institute for Advanced Studies(弗赖堡高级研究所);Xidian-FIAS International Joint Research Center(西电-弗赖堡国际联合研究中心);University of Texas at Austin(德克萨斯大学奥斯汀分校)
Human Gaze Boosts Object-Centered Representation Learning
Goethe University Frankfurt(法兰克福大学);The Hessian Center AI(黑森人工智能中心);Frankfurt Institute for Advanced Studies(法兰克福高等研究院);Xidian-FIAS international Joint Research Center(西安电子科技大学-法兰克福高等研究院国际联合研究中心)
Temporal Slowness in Central Vision Drives Semantic Object Learning
Department of Computer Science, Goethe University Frankfurt(法兰克福歌德大学计算机科学系);The Hessian Center for Artificial Intelligence (hessian.AI)(黑森人工智能中心);Frankfurt Institute for Advanced Studies(法兰克福高级研究 institute);Xidian-FIAS international Joint Research Center(西电-法兰克福国际联合研究中心)
Vision-Language Models Mistake Head Orientation for Gaze Direction: Nonverbal Conversation Cues
Brown University(布朗大学);Columbia University(哥伦比亚大学);Emory University(埃默里大学);Johns Hopkins University(约翰霍普金斯大学);University of Washington(华盛顿大学);Carnegie Mellon University(卡内基梅隆大学);University of Michigan(密歇根大学);UC San Diego(圣地亚哥大学)