In this paper, we focus on the multi-object tracking (MOT) problem of automatic driving and robot navigation. Most existing MOT methods track multiple objects using a singular RGB camera, which are prone to camera field-of-view and suffer tracking failures in complex scenarios due to background clutters and poor light conditions. To meet these challenges, we propose a MultiModality PAnoramic multi-object Tracking framework (MMPAT), which takes both 2D panorama images and 3D point clouds as input and then infers target trajectories using the multimodality data. The proposed method contains four major modules, a panorama image detection module, a multimodality data fusion module, a data association module and a trajectory inference model. We evaluate the proposed method on the JRDB dataset, where the MMPAT achieves the top performance in both the detection and tracking tasks and significantly outperforms state-of-the-art methods by a large margin (15.7 and 8.5 improvement in terms of AP and MOTA, respectively).
Nanjing University of Science and Technology（南京理工大学）
In this paper, we formulate a potentially valuable panoramic depth completion (PDC) task as panoramic 3D cameras often produce 360° depth with missing data in complex scenes. Its goal is to recover dense panoramic depths from raw sparse ones and panoramic RGB images. To deal with the PDC task, we train a deep network that takes both depth and image as inputs for the dense panoramic depth recovery. However, it needs to face a challenging optimization problem of the network parameters due to its non-convex objective function. To address this problem, we propose a simple yet effective approach te…
Federal University of Bahia（巴伊亚联邦大学）；Federal Institute of Bahia（巴伊亚联邦学院）；Southwest State University of Bahia（巴伊亚西南州立大学）
Deep learning has remarkably advanced in the last few years, supported by large labeled data sets. These data sets are precious yet scarce because of the time-consuming labeling procedures, discouraging researchers from producing them. This scarcity is especially true in dentistry, where deep learning applications are still in an embryonic stage. Motivated by this background, we address in this study the construction of a public data set of dental panoramic radiographs. Our objects of interest are the teeth, which are segmented and numbered, as they are the primary targets for dentists when sc…
Panoptic image segmentation is the computer vision task of finding groups of pixels in an image and assigning semantic classes and object instance identifiers to them. Research in image segmentation has become increasingly popular due to its critical applications in robotics and autonomous driving. The research community thereby relies on publicly available benchmark dataset to advance the state-of-the-art in computer vision. Due to the high costs of densely labeling the images, however, there is a shortage of publicly available ground truth labels that are suitable for panoptic segmentation. …
AInnovation Technology Co. Ltd（创新奇智科技有限公司）；Peking University（北京大学）；University of Washington（华盛顿大学）
Panoramic segmentation is a scene where image segmentation tasks is more difficult. With the development of CNN networks, panoramic segmentation tasks have been sufficiently developed.However, the current panoramic segmentation algorithms are more concerned with context semantics, but the details of image are not processed enough. Moreover, they cannot solve the problems which contains the accuracy of occluded object segmentation,little object segmentation,boundary pixel in object segmentation etc. Aiming to address these issues, this paper presents some useful tricks. (a) By changing the basi…
SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ARC Lab, Tencent PCG（腾讯平台与内容事业群ARC实验室）；Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院）；Zhejiang – Singapore Innovation and AI Joint Research Lab（浙江-新加坡创新与人工智能联合实验室）
A panoramic image mosaic is an attractive visualization for viewing many overlapping photos, but its images must be both captured and processed correctly to produce an acceptable composite. We propose Swipe Mosaics, an interactive visualization that places the individual video frames on a 2D planar map that represents the layout of the physical scene. Compared to traditional panoramic mosaics, our capture is easier because the user can both translate the camera center and film moving subjects. Processing and display degrade gracefully if the footage lacks distinct, overlapping, non-repeating t…
Image-based salient object detection (ISOD) in 360° scenarios is significant for understanding and applying panoramic information. However, research on 360° ISOD has not been widely explored due to the lack of large, complex, high-resolution, and well-labeled datasets. Towards this end, we construct a large scale 360° ISOD dataset with object-level pixel-wise annotation on equirectangular projection (ERP), which contains rich panoramic scenes with not less than 2K resolution and is the largest dataset for 360° ISOD by far to our best knowledge. By observing the data, we find current methods fa…
Salesforce Research Asia（Salesforce亚洲研究院）；Singapore Management University（新加坡管理大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局（A*STAR））
Designed for tracking user goals in dialogues, a dialogue state tracker is an essential component in a dialogue system. However, the research of dialogue state tracking has largely been limited to unimodality, in which slots and slot values are limited by knowledge domains (e.g. restaurant domain with slots of restaurant name and price range) and are defined by specific database schema. In this paper, we propose to extend the definition of dialogue state tracking to multimodality. Specifically, we introduce a novel dialogue state tracking task to track the information of visual objects that ar…
MICCAI STS 2024 Challenge: Semi-Supervised Instance-Level Tooth Segmentation in Panoramic X-ray and CBCT Images
Innovation Center for Electronic Design Automation Technology（电子设计自动化技术创新中心）；School of Cyberspace（网络空间学院）；Hangzhou Dianzi University（杭州电子科技大学）；Hangzhou Geriatric Stomatology Hospital（杭州老年牙科医院）；Shenzhen University（深圳大学）；Queen Mary University of London（伦敦大学Queen Mary分校）；Shandong University（山东大学）；Xidian University（西安电子科技大学）；Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology（哈尔滨工业大学）；The University of Hong Kong（香港大学）；Chinese Academy of Sciences（中国科学院）；Yunnan Provincial Stomatology Hospital（云南省牙科医院）；Shanghai MediWorks Precision Instruments Co., Ltd（上海 MediWorks 精密仪器有限公司）；University of Leicester（莱斯特大学）
Hunan University（湖南大学）；Hunan Normal University（湖南师范大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；HKUST(GZ)（香港科技大学（广州））；INSAIT, Sofia University “St. Kliment Ohridski”（索菲亚大学“圣克莱门特·奥赫里德斯基”INSAIT研究所）；Zhejiang University（浙江大学）
