Attributes of sound inherent to objects can provide valuable cues to learn rich representations for object detection and tracking. Furthermore, the co-occurrence of audiovisual events in videos can be exploited to localize objects over the image field by solely monitoring the sound in the environment. Thus far, this has only been feasible in scenarios where the camera is static and for single object detection. Moreover, the robustness of these methods has been limited as they primarily rely on RGB images which are highly susceptible to illumination and weather changes. In this work, we present the novel self-supervised MM-DistillNet framework consisting of multiple teachers that leverage diverse modalities including RGB, depth and thermal images, to simultaneously exploit complementary cues and distill knowledge into a single audio student network. We propose the new MTA loss function that facilitates the distillation of information from multimodal teachers in a self-supervised manner. Additionally, we propose a novel self-supervised pretext task for the audio student that enables us to not rely on labor-intensive manual annotations. We introduce a large-scale multimodal dataset with over 113,000 time-synchronized frames of RGB, depth, thermal, and audio modalities. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods while being able to detect multiple objects using only sound during inference and even while moving.
Knowledge distillation (KD) is a popular method to train efficient networks ("student") with the help of high-capacity networks ("teacher"). Traditional methods use the teacher's soft logits as extra supervision to train the student network. In this paper, we argue that it is more advantageous to make the student mimic the teacher's features in the penultimate layer. Not only the student can directly learn more effective information from the teacher feature, feature mimicking can also be applied for teachers trained without a softmax layer. Experiments show that it can achieve higher accuracy …
Ryerson University（瑞尔森大学）；Vector Institute for AI（向量人工智能研究所）；York University（约克大学）；Toronto AI Lab, LG Electronics（LG电子多伦多人工智能实验室）；Samsung AI Centre Toronto（三星多伦多人工智能中心）；University of Guelph（圭尔夫大学）
In this paper, we challenge the common assumption that collapsing the spatial dimensions of a 3D (spatial-channel) tensor in a convolutional neural network (CNN) into a vector via global pooling removes all spatial information. Specifically, we demonstrate that positional information is encoded based on the ordering of the channel dimensions, while semantic information is largely not. Following this demonstration, we show the real world impact of these findings by applying them to two applications. First, we propose a simple yet effective data augmentation strategy and loss function which impr…
A practical eye authentication (EA) system targeted for edge devices needs to perform authentication and be robust to presentation attacks, all while remaining compute and latency efficient. However, existing eye-based frameworks a) perform authentication and Presentation Attack Detection (PAD) independently and b) involve significant pre-processing steps to extract the iris region. Here, we introduce a joint framework for EA and PAD using periocular images. While a deep Multitask Learning (MTL) network can perform both the tasks, MTL suffers from the forgetting effect since the training datas…
Centre for Computational Intelligence (CCI)（计算智能中心）；De Montfort University（德蒙福特大学）；Fraunhofer HHI（弗劳恩霍夫海因里希·赫兹研究所）
A common heuristic in semi-supervised deep learning (SSDL) is to select unlabelled data based on a notion of semantic similarity to the labelled data. For example, labelled images of numbers should be paired with unlabelled images of numbers instead of, say, unlabelled images of cars. We refer to this practice as semantic data set matching. In this work, we demonstrate the limits of semantic data set matching. We show that it can sometimes even degrade the performance for a state of the art SSDL algorithm. We present and make available a comprehensive simulation sandbox, called non-IID-SSDL, f…
When Eye-Tracking Meets Machine Learning: A Systematic Review on Applications in Medical Image Analysis
Macquarie University（麦考瑞大学）；Macquarie Medical School（麦考瑞医学院）；Faculty of Medicine, Health and Human Sciences（医学与健康科学学院）；School of Computing（计算机学院）；Faculty of Science and Engineering（科学与工程学院）；Centre for Health Informatics（健康信息学中心）；Australian Institute of Health Innovation（澳大利亚健康创新研究院）
In the past few years, significant advancements were made in reconstruction of observed natural images from fMRI brain recordings using deep-learning tools. Here, for the first time, we show that dense 3D depth maps of observed 2D natural images can also be recovered directly from fMRI brain recordings. We use an off-the-shelf method to estimate the unknown depth maps of natural images. This is applied to both: (i) the small number of images presented to subjects in an fMRI scanner (images for which we have fMRI recordings - referred to as "paired" data), and (ii) a very large number of natura…
Eberhard Karls Universität Tübingen（图宾根大学）；Robert Bosch GmbH（罗伯特·博世有限公司）；University of Tübingen（图宾根大学）
After the introduction of smartphones and smartwatches, AR glasses are considered the next breakthrough in the field of wearables. While the transition from smartphones to smartwatches was based mainly on established display technologies, the display technology of AR glasses presents a technological challenge. Many display technologies, such as retina projectors, are based on continuous adaptive control of the display based on the user's pupil position. Furthermore, head-mounted systems require an adaptation and extension of established interaction concepts to provide the user with an immersiv…
This paper revisits recognition of natural image pleasantness by employing deep convolutional neural networks and affordable eye trackers. There exist several approaches to recognize image pleasantness: (1) computer vision, and (2) psychophysical signals. For natural images, computer vision approaches have not been as successful as for abstract paintings and is lagging behind the psychophysical signals like eye movements. Despite better results, the scalability of eye movements is adversely affected by the sensor cost. While the introduction of affordable sensors have helped the scalability is…
FACET: Fast and Accurate Event-Based Eye Tracking Using Ellipse Modeling for Extended Reality
School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院）；DVSense (Beijing) Technology Co., Ltd.（北京感观科技有限公司）；Delft University of Technology（代尔夫特理工大学）；Leiden Institute of Advanced Computer Science (LIACS), Leiden University（莱顿大学莱顿高级计算机科学研究所）
When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need
Sydney AI Centre, The University of Sydney（悉尼人工智能中心，悉尼大学）；School of Computer Science, Wuhan University（武汉大学计算机学院）；Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机系）
Nankai University（南开大学）；Beijing Shanggong Medical Technology Co. Ltd（北京上工医疗科技有限公司）
In ophthalmology, early fundus screening is an economic and effective way to prevent blindness caused by ophthalmic diseases. Clinically, due to the lack of medical resources, manual diagnosis is time-consuming and may delay the condition. With the development of deep learning, some researches on ophthalmic diseases have achieved good results, however, most of them are just based on one disease. During fundus screening, ophthalmologists usually give diagnoses of multi-disease on binocular fundus image, so we release a dataset with 8 diseases to meet the real medical scene, which contains 10,00…
Shanghai Jiao Tong University（上海交通大学）；the Second Affiliated Hospital of Zhejiang University, College of Medicine（浙江大学医学院附属第二医院）
The early diagnosis and screening of glaucoma are important for patients to receive treatment in time and maintain eyesight. Nowadays, deep learning (DL) based models have been successfully used for computer-aided diagnosis (CAD) of glaucoma from retina fundus images. However, a DL model pre-trained using a dataset from one hospital center may have poor performance on a dataset from another new hospital center and therefore its applications in the real scene are limited. In this paper, we propose a self-adaptive transfer learning (SATL) strategy to fill the domain gap between multicenter datas…
Drones Help Drones: A Collaborative Framework for Multi-Drone Object Trajectory Prediction and Beyond
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai AI Laboratory（上海人工智能实验室）
