We present a new method to localize a camera within a previously unseen environment perceived from an egocentric point of view. Although this is, in general, an ill-posed problem, humans can effortlessly and efficiently determine their relative location and orientation and navigate into a previously unseen environments, e.g., finding a specific item in a new grocery store. To enable such a capability, we design a new egocentric representation, which we call ECO (Egocentric COgnitive map). ECO is biologically inspired, by the cognitive map that allows human navigation, and it encodes the surrounding visual semantics with respect to both distance and orientation. ECO possesses three main properties: (1) reconfigurability: complex semantics and geometry is captured via the synthesis of atomic visual representations (e.g., image patch); (2) robustness: the visual semantics are registered in a geometrically consistent way (e.g., aligning with respect to the gravity vector, frontalizing, and rescaling to canonical depth), thus enabling us to learn meaningful atomic representations; (3) adaptability: a domain adaptation framework is designed to generalize the learned representation without manual calibration. As a proof-of-concept, we use ECO to localize a camera within real-world scenes---various grocery stores---and demonstrate performance improvements when compared to existing semantic localization approaches.
Universitat de Barcelona（巴塞罗那大学）；University of Groningen（格罗宁根大学）；Consorci Sanitari de Terrassa（特拉萨医疗联盟）
Mild cognitive impairment is the early stage of several neurodegenerative diseases, such as Alzheimer's. In this work, we address the use of lifelogging as a tool to obtain pictures from a patient's daily life from an egocentric point of view. We propose to use them in combination with serious games as a way to provide a non-pharmacological treatment to improve their quality of life. To do so, we introduce a novel computer vision technique that classifies rich and non rich egocentric images and uses them in serious games. We present results over a dataset composed by 10,997 images, recorded by…
Egocentric perception has grown rapidly with the advent of immersive computing devices. Human gaze prediction is an important problem in analyzing egocentric videos and has primarily been tackled through either saliency-based modeling or highly supervised learning. We quantitatively analyze the generalization capabilities of supervised, deep learning models on the egocentric gaze prediction task on unseen, out-of-domain data. We find that their performance is highly dependent on the training data and is restricted to the domains specified in the training annotations. In this work, we tackle th…
EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
Tsinghua University（清华大学）；Tongji University（同济大学）；Renmin University of China（中国人民大学）；The University of Tokyo（东京大学）；Lenovo Group（联想集团）；Peking University（北京大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Shanghai Qi Zhi Institute（上海启智研究院）
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Fraunhofer IPK（弗劳恩霍夫研究所）；Technical University of Berlin（柏林技术大学）；University of Tübingen（图宾根大学）；RWTH Aachen University（亚琛工业大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
OmniView-Space: Reinforcing Spatial Reasoning via Multi-Perspective Spatial Mapping
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（厦门大学多媒体可信感知与高效计算教育部重点实验室）；Tencent Youtu Lab（腾讯优图实验室）；Beijing Institute of Technology（北京理工大学）
CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；King Abdullah University of Science and Technology（科廷大学）；Fudan University（复旦大学）
Cognitive Science Program, University of California, Berkeley（加州大学伯克利分校认知科学项目）；Department of Electrical and Computer Engineering, University of California San Diego（加州大学圣地亚哥分校电气与计算机工程系）；School of Computer Science, Georgia Institute of Technology & Emory University（佐治亚理工学院计算机科学学院及埃默里大学）；Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）；Department of Cognitive Science, University of California San Diego（加州大学圣地亚哥分校认知科学系）；Equal Advising Department of Computer Science & Wilmer Eye Institute, Johns Hopkins University（约翰霍普金斯大学计算机科学系及威尔默眼科研究所）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Weinberg Institute for Cognitive Science, University of Michigan（密歇根大学韦恩伯格认知科学研究所）
