We present the Multiview Extended Video with Activities (MEVA) dataset, a new and very-large-scale dataset for human activity recognition. Existing security datasets either focus on activity counts by aggregating public video disseminated due to its content, which typically excludes same-scene background video, or they achieve persistence by observing public areas and thus cannot control for activity content. Our dataset is over 9300 hours of untrimmed, continuous video, scripted to include diverse, simultaneous activities, along with spontaneous background activity. We have annotated 144 hours for 37 activity types, marking bounding boxes of actors and props. Our collection observed approximately 100 actors performing scripted scenarios and spontaneous background activity over a three-week period at an access-controlled venue, collecting in multiple modalities with overlapping and non-overlapping indoor and outdoor viewpoints. The resulting data includes video from 38 RGB and thermal IR cameras, 42 hours of UAV footage, as well as GPS locations for the actors. 122 hours of annotation are sequestered in support of the NIST Activity in Extended Video (ActEV) challenge; the other 22 hours of annotation and the corresponding video are available on our website, along with an additional 306 hours of ground camera data, 4.6 hours of UAV data, and 9.6 hours of GPS logs. Additional derived data includes camera models geo-registering the outdoor cameras and a dense 3D point cloud model of the outdoor scene. The data was collected with IRB oversight and approval and released under a CC-BY-4.0 license.
This report describes the approach behind our winning solution to the 2022 Epic-Kitchens Action Recognition Challenge. Our approach builds upon our recent work, Multiview Transformer for Video Recognition (MTV), and adapts it to multimodal inputs. Our final submission consists of an ensemble of Multimodal MTV (M&M) models varying backbone sizes and input modalities. Our approach achieved 52.8% Top-1 accuracy on the test set in action classes, which is 4.1% higher than last year's winning entry.
Argus: Leveraging Multiview Images for Improved 3-D Scene Understanding With Large Language Models
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Beijing Digital Native Digital City Research Center（北京数字原生数字城市研究院）；School of Computing, The University of Georgia（佐治亚大学计算机学院）；School of Computer and Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学（深圳）计算机科学与工程系）
Univ. of California Santa Barbara（加州大学圣巴巴拉分校）；West Virginia University（西弗吉尼亚大学）；Santa Barbara Cottage Hospital（圣巴巴拉 Cottage 医院）
Manual analysis of body poses of bed-ridden patients requires staff to continuously track and record patient poses. Two limitations in the dissemination of pose-related therapies are scarce human resources and unreliable automated systems. This work addresses these issues by introducing a new method and a new system for robust automated classification of sleep poses in an Intensive Care Unit (ICU) environment. The new method, coupled-constrained Least-Squares (cc-LS), uses multimodal and multiview (MM) data and finds the set of modality trust values that minimizes the difference between expect…
Guangdong Polytechnic Normal University（广东技术师范大学）；Guangdong University of Technology（广东工业大学）
In deep learning area, large-scale image datasets bring a breakthrough in the success of object recognition and retrieval. Nowadays, as the embodiment of innovation, the diversity of the industrial goods is significantly larger, in which the incomplete multiview, multimodal and multilabel are different from the traditional dataset. In this paper, we introduce an industrial goods dataset, namely PatentNet, with numerous highly diverse, accurate and detailed annotations of industrial goods images, and corresponding texts. In PatentNet, the images and texts are sourced from design patent. Within …
FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment
School of Biomedical Engineering (Suzhou), USTC（中国科学技术大学苏州生物医学工程学院）；Suzhou Institute of Biomedical Engineering and Technology, CAS（中国科学院苏州生物医学工程技术研究所）；Institute of High-Performance Computing, A*STAR, Singapore（新加坡科技研究局高性能计算研究所）；School of Psychology, Beijing Sports University（北京体育大学心理学院）；School of Competitive Sports, Beijing Sports University（北京体育大学竞技体育学院）；Department of Robotics, University of Michigan（密歇根大学机器人系）
University of California Santa Barbara（加州大学圣巴巴拉分校）；Santa Barbara Cottage Hospital（圣巴巴拉村舍医院）
Clinical observations indicate that during critical care at the hospitals, patients sleep positioning and motion affect recovery. Unfortunately, there is no formal medical protocol to record, quantify, and analyze patient motion. There is a small number of clinical studies, which use manual analysis of sleep poses and motion recordings to support medical benefits of patient positioning and motion monitoring. Manual processes are not scalable, are prone to human errors, and strain an already taxed healthcare workforce. This study introduces DECU (Deep Eye-CU): an autonomous mulitmodal multiview…
MVGGT: Multimodal Visual Geometry Grounded Transformer for Multiview 3D Referring Expression Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（厦门大学多媒体可信感知与高效计算教育部重点实验室）；Shanghai Innovation Institute（上海创新研究院）；Fudan University（复旦大学）；ByteDance（字节跳动）；Tianjin University of Science and Technology（天津科技大学）
University of Liège（列日大学）；Aalborg University（奥尔堡大学）
Monitoring the occupancy of public sports facilities is essential to assess their use and to motivate their construction in new places. In the case of a football field, the area to cover is large, thus several regular cameras should be used, which makes the setup expensive and complex. As an alternative, we developed a system that detects players from a unique cheap and wide-angle fisheye camera assisted by a single narrow-angle thermal camera. In this work, we train a network in a knowledge distillation approach in which the student and the teacher have different modalities and a different vi…
Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；Institut de Robòtica i Informàtica Industrial, CSIC-UPC（工业机器人与信息学研究所，CSIC-UPC）；Carnegie Mellon University（卡内基梅隆大学）；Facebook AI；Gallaudet University（加劳德特大学）
One of the factors that have hindered progress in the areas of sign language recognition, translation, and production is the absence of large annotated datasets. Towards this end, we introduce How2Sign, a multimodal and multiview continuous American Sign Language (ASL) dataset, consisting of a parallel corpus of more than 80 hours of sign language videos and a set of corresponding modalities including speech, English transcripts, and depth. A three-hour subset was further recorded in the Panoptic studio enabling detailed 3D pose estimation. To evaluate the potential of How2Sign for real-world …
Johns Hopkins University（约翰斯·霍普金斯大学）；Carnegie Mellon University（卡内基梅隆大学）；Imperial College London（伦敦帝国理工学院）
An increasing number of datasets contain multiple views, such as video, sound and automatic captions. A basic challenge in representation learning is how to leverage multiple views to learn better representations. This is further complicated by the existence of a latent alignment between views, such as between speech and its transcription, and by the multitude of choices for the learning objective. We explore an advanced, correlation-based representation learning method on a 4-way parallel, multimodal dataset, and assess the quality of the learned representations on retrieval-based tasks. We s…
Explaining Recovery Trajectories of Older Adults Post Lower-Limb Fracture Using Modality-wise Multiview Clustering and Large Language Models
KITE Research Institute（KITE研究机构）；Toronto Rehabilitation Institute（多伦多康复研究所）；University Health Network（大学健康网络）；College of Engineering and Technology（工程与技术学院）
Center for Research in Computer Vision（计算机视觉研究中心）；University of Central Florida（中佛罗里达大学）
Activity detection in security videos is a difficult problem due to multiple factors such as large field of view, presence of multiple activities, varying scales and viewpoints, and its untrimmed nature. The existing research in activity detection is mainly focused on datasets, such as UCF-101, JHMDB, THUMOS, and AVA, which partially address these issues. The requirement of processing the security videos in real-time makes this even more challenging. In this work we propose Gabriella, a real-time online system to perform activity detection on untrimmed security videos. The proposed method cons…
Human Activity Recognition is a field of research where input data can take many forms. Each of the possible input modalities describes human behaviour in a different way, and each has its own strengths and weaknesses. We explore the hypothesis that leveraging multiple modalities can lead to better recognition. Since manual annotation of input data is expensive and time-consuming, the emphasis is made on self-supervised methods which can learn useful feature representations without any ground truth labels. We extend a number of recent contrastive self-supervised approaches for the task of Huma…
FlexLoc: Conditional Neural Networks for Zero-Shot Sensor Perspective Invariance in Object Localization with Distributed Multimodal Sensors
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；US DEVCOM Army Research Laboratory（美国DEVCOM陆军研究实验室）；Amazon（亚马逊）
MyoMechanix: Biomechanically-Grounded Compositional Skilled Activity Understanding and Coaching
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences（中国科学院苏州生物医学工程技术研究所）；Arexeni Research and Technologies Inc.（Arexeni研究与技术公司）；Beijing Sports University（北京体育大学）；University of Michigan（密歇根大学）
The data-driven approach that learns an optimal representation of vision features like skeleton frames or RGB videos is currently a dominant paradigm for activity recognition. While great improvements have been achieved from existing single modal approaches with increasingly larger datasets, the fusion of various data modalities at the feature level has seldom been attempted. In this paper, we propose a multimodal feature fusion model that utilizes both skeleton and RGB modalities to infer human activity. The objective is to improve the activity recognition accuracy by effectively utilizing th…
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
FAIR, Meta（Meta FAIR实验室）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）；Project Aria, Meta（Meta Project Aria）；University of Bristol（布里斯托尔大学）；IIIT, Hyderabad（印度信息技术研究所海得拉巴分校）；University of Illinois, Urbana Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Minnesota（明尼苏达大学）；National University of Singapore（新加坡国立大学）；Universidad de los Andes（安第斯大学）；Simon Fraser University（西蒙菲莎大学）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
