Multi-camera tracking systems are gaining popularity in applications that demand high-quality tracking results, such as frictionless checkout because monocular multi-object tracking (MOT) systems often fail in cluttered and crowded environments due to occlusion. Multiple highly overlapped cameras can significantly alleviate the problem by recovering partial 3D information. However, the cost of creating a high-quality multi-camera tracking dataset with diverse camera settings and backgrounds has limited the dataset scale in this domain. In this paper, we provide a large-scale densely-labeled multi-camera tracking dataset in five different environments with the help of an auto-annotation system. The system uses overlapped and calibrated depth and RGB cameras to build a high-performance 3D tracker that automatically generates the 3D tracking results. The 3D tracking results are projected to each RGB camera view using camera parameters to create 2D tracking results. Then, we manually check and correct the 3D tracking results to ensure the label quality, which is much cheaper than fully manual annotation. We have conducted extensive experiments using two real-time multi-camera trackers and a person re-identification (ReID) model with different settings. This dataset provides a more reliable benchmark of multi-camera, multi-object tracking systems in cluttered and crowded environments. Also, our results demonstrate that adapting the trackers and ReID models on this dataset significantly improves their performance. Our dataset will be publicly released upon the acceptance of this work.
TEMSET-24K: Densely Annotated Dataset for Indexing Multipart Endoscopic Videos using Surgical Timeline Segmentation
Birmingham City University（伯明翰城市大学）；University Hospitals Birmingham（伯明翰大学医院）；University of Birmingham（伯明翰大学）；University of the West of England（西英格兰大学）；University of Hertfordshire（赫特福德大学）；University of Bristol（布里斯托大学）；Qatar University（卡塔尔大学）
For the deployment of artificial intelligence (AI) in high-risk settings, such as healthcare, methods that provide interpretability/explainability or allow fine-grained error analysis are critical. Many recent methods for interpretability/explainability and fine-grained error analysis use concepts, which are meta-labels that are semantically meaningful to humans. However, there are only a few datasets that include concept-level meta-labels and most of these meta-labels are relevant for natural images that do not require domain expertise. Densely annotated datasets in medicine focused on meta-l…
Object grasping is critical for many applications, which is also a challenging computer vision problem. However, for the clustered scene, current researches suffer from the problems of insufficient training data and the lacking of evaluation benchmarks. In this work, we contribute a large-scale grasp pose detection dataset with a unified evaluation system. Our dataset contains 87,040 RGBD images with over 370 million grasp poses. Meanwhile, our evaluation system directly reports whether a grasping is successful or not by analytic computation, which is able to evaluate any kind of grasp poses w…
EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models
École Polytechnique Fédérale de Lausanne (EPFL)（瑞士联邦理工学院洛桑校区）；Microsoft（微软）；Scuola Superiore Sant’Anna（圣安娜高等学院）；Swiss Federal Institute of Technology Valais (EPFL Valais)（瑞士联邦技术学院瓦莱分校）；University of Geneva Medical School（日内瓦大学医学院）；Eidgenössische Technische Hochschule (ETH)（瑞士联邦理工学院）
This paper introduces DensePoint, a densely sampled and annotated point cloud dataset containing over 10,000 single objects across 16 categories, by merging different kind of information from two existing datasets. Each point cloud in DensePoint contains 40,000 points, and each point is associated with two sorts of information: RGB value and part annotation. In addition, we propose a method for point cloud colorization by utilizing Generative Adversarial Networks (GANs). The network makes it possible to generate colours for point clouds of single objects by only giving the point cloud itself. …
We present a robust method for estimating the facial pose and shape information from a densely annotated facial image. The method relies on Convolutional Point-set Representation (CPR), a carefully designed matrix representation to summarize different layers of information encoded in the set of detected points in the annotated image. The CPR disentangles the dependencies of shape and different pose parameters and enables updating different parameters in a sequential manner via convolutional neural networks and recurrent layers. When updating the pose parameters, we sample reprojection errors a…
CNeuroMod-THINGS, a densely-sampled fMRI dataset for visual neuroscience
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；Centre de recherche de l’Institut universitaire de gériatrie de Montréal（蒙特利尔大学老年医学研究中心）；Université de Montréal（蒙特利尔大学）；Martin Luther University Halle-Wittenberg（马丁·路德大学哈勒-维滕贝格）；Université de Genève（日内瓦大学）；Justus Liebig University Giessen（吉森尤斯图斯·李比希大学）；Center for Mind, Brain and Behavior (CMBB), Universities of Marburg, Giessen, and Darmstadt（心智、大脑与行为中心（马尔堡大学、吉森大学和达姆施塔特工业大学））
RGBT tracking receives a surge of interest in the computer vision community, but this research field lacks a large-scale and high-diversity benchmark dataset, which is essential for both the training of deep RGBT trackers and the comprehensive evaluation of RGBT tracking methods. To this end, we present a Large-scale High-diversity benchmark for RGBT tracking (LasHeR) in this work. LasHeR consists of 1224 visible and thermal infrared video pairs with more than 730K frame pairs in total. Each frame pair is spatially aligned and manually annotated with a bounding box, making the dataset well and…
University of Bristol（布里斯托大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；NVIDIA（英伟达）；University of Catania（卡塔尼亚大学）
Since its introduction in 2018, EPIC-KITCHENS has attracted attention as the largest egocentric video benchmark, offering a unique viewpoint on people's interaction with objects, their attention, and even intention. In this paper, we detail how this large-scale dataset was captured by 32 participants in their native kitchen environments, and densely annotated with actions and object interactions. Our videos depict nonscripted daily activities, as recording is started every time a participant entered their kitchen. Recording took place in 4 countries by participants belonging to 10 different na…
Supervised learning is often used to count objects in images, but for counting small, densely located objects, the required image annotations are burdensome to collect. Counting plant organs for image-based plant phenotyping falls within this category. Object counting in plant images is further challenged by having plant image datasets with significant domain shift due to different experimental conditions, e.g. applying an annotated dataset of indoor plant images for use on outdoor images, or on a different plant species. In this paper, we propose a domain-adversarial learning approach for dom…
The University of Adelaide（阿德莱德大学）；Samsung AI Cambridge（三星剑桥人工智能中心）；Stanford University（斯坦福大学）；Monash University（莫纳什大学）
The availability of large-scale video action understanding datasets has facilitated advances in the interpretation of visual scenes containing people. However, learning to recognise human actions and their social interactions in an unconstrained real-world environment comprising numerous people, with potentially highly unbalanced and long-tailed distributed action labels from a stream of sensory data captured from a mobile robot platform remains a significant challenge, not least owing to the lack of a reflective large-scale dataset. In this paper, we introduce JRDB-Act, as an extension of the…
Tree perception is an essential building block toward autonomous forestry operations. Current developments generally consider input data from lidar sensors to solve forest navigation, tree detection and diameter estimation problems. Whereas cameras paired with deep learning algorithms usually address species classification or forest anomaly detection. In either of these cases, data unavailability and forest diversity restrain deep learning developments for autonomous systems. So, we propose two densely annotated image datasets - 43k synthetic, 100 real - for bounding box, segmentation mask and…
Video analysis has been moving towards more detailed interpretation (e.g. segmentation) with encouraging progresses. These tasks, however, increasingly rely on densely annotated training data both in space and time. Since such annotation is labour-intensive, few densely annotated video data with detailed region boundaries exist. This work aims to resolve this dilemma by learning to automatically generate region boundaries for all frames of a video from sparsely annotated bounding boxes of target regions. We achieve this with a Volumetric Graph Convolutional Network (VGCN), which learns to iter…
Visual Objectification in Films: Towards a New AI Task for Video Interpretation
Université Côte d’Azur（蔚蓝海岸大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）；Institut Universitaire de France（法国大学研究院）；Université Sorbonne Paris Nord（巴黎第十三大学）；LabSIC（信息与传播科学实验室）；Sorbonne Université（索邦大学）；GRIPIC（信息与传播研究与教学组）；BCL（巴斯克语言与文化实验室）
Georgia Institute of Technology（佐治亚理工学院）；University of Michigan（密歇根大学）
Missions to small celestial bodies rely heavily on optical feature tracking for characterization of and relative navigation around the target body. While deep learning has led to great advancements in feature detection and description, training and validating data-driven models for space applications is challenging due to the limited availability of large-scale, annotated datasets. This paper introduces AstroVision, a large-scale dataset comprised of 115,970 densely annotated, real images of 16 different small bodies captured during past and ongoing missions. We leverage AstroVision to develop…
Towards All-Day Perception for Off-Road Driving: A Large-Scale Multispectral Dataset and Comprehensive Benchmark
Research Center for Intelligent Computing Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing（智能计算系统研究中心，计算技术研究所，中国科学院，北京）；Beijing Institute of Control Engineering（北京控制工程研究所）
Massachusetts Institute of Technology（麻省理工学院）；Dartmouth College（达特茅斯学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
This paper presents a new large-scale dataset for recognition and temporal localization of human actions collected from Web videos. We refer to it as HACS (Human Action Clips and Segments). We leverage both consensus and disagreement among visual classifiers to automatically mine candidate short clips from unlabeled videos, which are subsequently validated by human annotators. The resulting dataset is dubbed HACS Clips. Through a separate process we also collect annotations defining action segment boundaries. This resulting dataset is called HACS Segments. Overall, HACS Clips consists of 1.5M …
Universidad de los Andes（安第斯大学）；Adobe Research（Adobe研究院）；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）
Humans are arguably one of the most important subjects in video streams, many real-world applications such as video summarization or video editing workflows often require the automatic search and retrieval of a person of interest. Despite tremendous efforts in the person reidentification and retrieval domains, few works have developed audiovisual search strategies. In this paper, we present the Audiovisual Person Search dataset (APES), a new dataset composed of untrimmed videos whose audio (voices) and visual (faces) streams are densely annotated. APES contains over 1.9K identities labeled alo…
For further progress in video object segmentation (VOS), larger, more diverse, and more challenging datasets will be necessary. However, densely labeling every frame with pixel masks does not scale to large datasets. We use a deep convolutional network to automatically create pseudo-labels on a pixel level from much cheaper bounding box annotations and investigate how far such pseudo-labels can carry us for training state-of-the-art VOS approaches. A very encouraging result of our study is that adding a manually annotated mask in only a single video frame for each object is sufficient to gener…
Advanced Research and Innovation Center (ARIC), Khalifa University（哈利法大学高级研究与创新中心（ARIC））；Khalifa University Center for Autonomous Robotic Systems (KUCARS), Khalifa University（哈利法大学自主机器人系统中心（KUCARS））；School of Computer Science and Mathematics, Kingston University（金斯顿大学计算机科学与数学学院）；Department of Aerospace Engineering, Khalifa University（哈利法大学航空航天工程系）
Taking advantage of an event-based camera, the issues of motion blur, low dynamic range and low time sampling of standard cameras can all be addressed. However, there is a lack of event-based datasets dedicated to the benchmarking of segmentation algorithms, especially those that provide depth information which is critical for segmentation in occluded scenes. This paper proposes a new Event-based Segmentation Dataset (ESD), a high-quality 3D spatial and temporal dataset for object segmentation in an indoor cluttered environment. Our proposed dataset ESD comprises 145 sequences with 14,166 RGB …
WBCAtt+: Fine-Grained Pixel-Level Morphological Annotations for White Blood Cell Images
Rapid-Rich Object Search (ROSE) Lab, School of Electrical and Electronic Engineering, Nanyang Technological University（快速丰富目标搜索（ROSE）实验室，电气与电子工程学院，南洋理工大学）；Shanghai University of Finance and Economics（上海财经大学）
Beijing Institute of Technology（北京理工大学）；German Aerospace Center (DLR)（德国航空航天中心（DLR））；Mississippi State University（密西西比州立大学）
Vehicle detection in remote sensing images has attracted increasing interest in recent years. However, its detection ability is limited due to lack of well-annotated samples, especially in densely crowded scenes. Furthermore, since a list of remotely sensed data sources is available, efficient exploitation of useful information from multi-source data for better vehicle detection is challenging. To solve the above issues, a multi-source active fine-tuning vehicle detection (Ms-AFt) framework is proposed, which integrates transfer learning, segmentation, and active classification into a unified …
Beihang University（北京航空航天大学）；Sea AI Lab（Sea AI实验室）
Vision and language understanding techniques have achieved remarkable progress, but currently it is still difficult to well handle problems involving very fine-grained details. For example, when the robot is told to "bring me the book in the girl's left hand", most existing methods would fail if the girl holds one book respectively in her left and right hand. In this work, we introduce a new task named human-centric relation segmentation (HRS), as a fine-grained case of HOI-det. HRS aims to predict the relations between the human and surrounding entities and identify the relation-correlated hu…
University of Edinburgh（爱丁堡大学）；Nanyang Technological University（南洋理工大学）；Shanghai AI Laboratory（上海人工智能实验室）
Generative models for audio-conditioned dance motion synthesis map music features to dance movements. Models are trained to associate motion patterns to audio patterns, usually without an explicit knowledge of the human body. This approach relies on a few assumptions: strong music-dance correlation, controlled motion data and relatively simple poses and movements. These characteristics are found in all existing datasets for dance motion synthesis, and indeed recent methods can achieve good results.We introduce a new dataset aiming to challenge these common assumptions, compiling a set of dynam…
FunnyNodules: A Customizable Medical Dataset Tailored for Evaluating Explainable AI
Ulm University Medical Center（乌尔姆大学医学中心）；XAIRAD - Cooperation for Artificial Intelligence in Experimental Radiology（XAIRAD——实验放射学人工智能合作）；Department of Diagnostic and Interventional Radiology（诊断与介入放射学系）
Institute of Automation, CAS（中国科学院自动化研究所）；School of Artificial Intelligence, UCAS（中国科学院大学人工智能学院）；Youtu Lab, Tencent（腾讯优图实验室）；CASIA-LLVision Joint Lab（中国科学院自动化研究所-联汇视界联合实验室）；Y-Tech, Kuaishou Technology（快手科技Y-Tech）
Object detection has achieved remarkable progress in the past decade. However, the detection of oriented and densely packed objects remains challenging because of following inherent reasons: (1) receptive fields of neurons are all axis-aligned and of the same shape, whereas objects are usually of diverse shapes and align along various directions; (2) detection models are typically trained with generic knowledge and may not generalize well to handle specific objects at test time; (3) the limited dataset hinders the development on this task. To resolve the first two issues, we present a dynamic …
HHMI Janelia Research Campus（霍华德·休斯医学研究所珍利亚研究园区）；Institute of Neuroinformatics UZH/ETHZ（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）
We present a method for microtubule tracking in electron microscopy volumes. Our method first identifies a sparse set of voxels that likely belong to microtubules. Similar to prior work, we then enumerate potential edges between these voxels, which we represent in a candidate graph. Tracks of microtubules are found by selecting nodes and edges in the candidate graph by solving a constrained optimization problem incorporating biological priors on microtubule structure. For this, we present a novel integer linear programming formulation, which results in speed-ups of three orders of magnitude an…
We introduce a new large-scale data set of video URLs with densely-sampled object bounding box annotations called YouTube-BoundingBoxes (YT-BB). The data set consists of approximately 380,000 video segments about 19s long, automatically selected to feature objects in natural settings without editing or post-processing, with a recording quality often akin to that of a hand-held cell phone camera. The objects represent a subset of the MS COCO label set. All video segments were human-annotated with high-precision classification labels and bounding boxes at 1 frame per second. The use of a cascade…
The SA-FARI Dataset: Segment Anything in Footage of Animals for Recognition and Identification
Conservation X Labs (CXL)（Conservation X Labs）；Meta；University of Bristol（布里斯托大学）；Hasso Plattner Institute（哈索罗滕堡研究所）；University of Oviedo（奥维多大学）；Osa Conservation（Osa保护）；Senckenberg Museum of Natural History（Senckenberg自然史博物馆）；Max Planck Institute for Evolutionary Anthropology（马克斯·普朗克进化人类学研究所）；Climate Corridors（气候走廊）
AnyStar: Domain randomized universal star-convex 3D instance segmentation
MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Boston Children’s Hospital（波士顿儿童医院）；Harvard Medical School（哈佛医学院）；Martinos Center（马蒂诺斯中心）；Massachusetts General Hospital（麻省总医院）
A New Dataset and Performance Benchmark for Real-time Spacecraft Segmentation in Onboard Computers
Department of Computer Science, Rice University（计算机科学系，里士大学）；Johnson Space Center, NASA（约翰逊航天中心，美国国家航空航天局）；Data to Knowledge Lab, Rice University（数据到知识实验室，里士大学）
Object proposals have quickly become the de-facto pre-processing step in a number of vision pipelines (for object detection, object discovery, and other tasks). Their performance is usually evaluated on partially annotated datasets. In this paper, we argue that the choice of using a partially annotated dataset for evaluation of object proposals is problematic -- as we demonstrate via a thought experiment, the evaluation protocol is 'gameable', in the sense that progress under this protocol does not necessarily correspond to a "better" category independent object proposal algorithm. To alleviat…
Are Dense Labels Always Necessary for 3D Object Detection from Point Cloud?
School of Intelligent Systems Engineering, the Shenzhen Campus of Sun Yatsen University（南方科技大学深圳校区智能系统工程学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Mathematics and Statistics and Ministry of Education Key Lab of Intelligent Networks and Network Security, Xi’an Jiaotong University（西安交通大学数学与统计学系和教育部智能网络与网络安全重点实验室）；School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications（重庆邮电大学通信与信息工程学院）；Meta；University of Maryland（美国马里兰大学）
Existing state-of-the-art methods for Video Object Segmentation (VOS) learn low-level pixel-to-pixel correspondences between frames to propagate object masks across video. This requires a large amount of densely annotated video data, which is costly to annotate, and largely redundant since frames within a video are highly correlated. In light of this, we propose HODOR: a novel method that tackles VOS by effectively leveraging annotated static images for understanding object appearance and scene context. We encode object instances and scene information from an image frame into robust high-level…
Trinity College Dublin（都柏林圣三一学院）；University of Houston-Victoria（休斯顿大学维多利亚分校）；Tarbiat Modares University（塔比亚特·莫达雷斯大学）
Scene understanding of full-scale 3D models of an urban area remains a challenging task. While advanced computer vision techniques offer cost-effective approaches to analyse 3D urban elements, a precise and densely labelled dataset is quintessential. The paper presents the first-ever labelled dataset for a highly dense Aerial Laser Scanning (ALS) point cloud at city-scale. This work introduces a novel benchmark dataset that includes a manually annotated point cloud for over 260 million laser scanning points into 100'000 (approx.) assets from Dublin LiDAR point cloud [12] in 2015. Objects are l…
Temple University（天普大学）；South China Univ. of Tech.（华南理工大学）；Peng Cheng Laboratory（鹏城实验室）；Meitu HiScene Lab, HiScene Information Technologies（美图HiScene实验室、HiScene信息技术公司）
In this paper, we present LaSOT, a high-quality benchmark for Large-scale Single Object Tracking. LaSOT consists of 1,400 sequences with more than 3.5M frames in total. Each frame in these sequences is carefully and manually annotated with a bounding box, making LaSOT the largest, to the best of our knowledge, densely annotated tracking benchmark. The average video length of LaSOT is more than 2,500 frames, and each sequence comprises various challenges deriving from the wild where target objects may disappear and re-appear again in the view. By releasing LaSOT, we expect to provide the commun…
University of Udine（乌迪内大学）；University of Catania（卡塔尼亚大学）
Understanding human-object interactions is fundamental in First Person Vision (FPV). Tracking algorithms which follow the objects manipulated by the camera wearer can provide useful cues to effectively model such interactions. Despite a few previous attempts to exploit trackers in FPV applications, a methodical analysis of the performance of state-of-the-art visual trackers in this domain is still missing. In this short paper, we provide a recap of the first systematic study of object tracking in FPV. Our work extensively analyses the performance of recent and baseline FPV trackers with respec…
Stony Brook University（石溪大学）；Temple University（天普大学）；South China University of Technology（华南理工大学）；Peng Cheng Laboratory（鹏城实验室）；HiScene Information Technologies（亮风台信息科技有限公司）；Amazon Web Services（亚马逊云科技）
Despite great recent advances in visual tracking, its further development, including both algorithm design and evaluation, is limited due to lack of dedicated large-scale benchmarks. To address this problem, we present LaSOT, a high-quality Large-scale Single Object Tracking benchmark. LaSOT contains a diverse selection of 85 object classes, and offers 1,550 totaling more than 3.87 million frames. Each video frame is carefully and manually annotated with a bounding box. This makes LaSOT, to our knowledge, the largest densely annotated tracking benchmark. Our goal in releasing LaSOT is to provi…
University of Udine（乌迪内大学）；University of Catania（卡塔尼亚大学）
Understanding human-object interactions is fundamental in First Person Vision (FPV). Tracking algorithms which follow the objects manipulated by the camera wearer can provide useful cues to effectively model such interactions. Visual tracking solutions available in the computer vision literature have significantly improved their performance in the last years for a large variety of target objects and tracking scenarios. However, despite a few previous attempts to exploit trackers in FPV applications, a methodical analysis of the performance of state-of-the-art trackers in this domain is still m…
