Existing methods of privacy-preserving action recognition (PPAR) mainly focus on frame-level (spatial) privacy removal through 2D CNNs. Unfortunately, they have two major drawbacks. First, they may compromise temporal dynamics in input videos, which are critical for accurate action recognition. Second, they are vulnerable to practical attacking scenarios where attackers probe for privacy from an entire video rather than individual frames. To address these issues, we propose a novel framework STPrivacy to perform video-level PPAR. For the first time, we introduce vision Transformers into PPAR by treating a video as a tubelet sequence, and accordingly design two complementary mechanisms, i.e., sparsification and anonymization, to remove privacy from a spatio-temporal perspective. In specific, our privacy sparsification mechanism applies adaptive token selection to abandon action-irrelevant tubelets. Then, our anonymization mechanism implicitly manipulates the remaining action-tubelets to erase privacy in the embedding space through adversarial learning. These mechanisms provide significant advantages in terms of privacy preservation for human eyes and action-privacy trade-off adjustment during deployment. We additionally contribute the first two large-scale PPAR benchmarks, VP-HMDB51 and VP-UCF101, to the community. Extensive evaluations on them, as well as two other tasks, validate the effectiveness and generalization capability of our framework.
From Pixels to Privacy: Temporally Consistent Video Anonymization via Token Pruning for Privacy Preserving Action Recognition
Aalborg University（奥尔堡大学）；Aarhus University（奥胡斯大学）；Poineer Centre for AI（先锋人工智能中心）；University of Southern Denmark（南丹麦大学）；Milestone System（Milestone系统公司）
Indiana University, Bloomington（印第安纳大学伯明顿分校）；University of Central Florida（中佛罗里达大学）；Midea Corporate Research Center（美的企业研究中心）
A major emerging challenge is how to protect people's privacy as cameras and computer vision are increasingly integrated into our daily lives, including in smart devices inside homes. A potential solution is to capture and record just the minimum amount of information needed to perform a task of interest. In this paper, we propose a fully-coupled two-stream spatiotemporal architecture for reliable human action recognition on extremely low resolution (e.g., 12x16 pixel) videos. We provide an efficient method to extract spatial and temporal features and to aggregate them into a robust feature re…
Human actions recognition is a fundamental task in artificial vision, that has earned a great importance in recent years due to its multiple applications in different areas. %, such as the study of human behavior, security or video surveillance. In this context, this paper describes an approach for real-time human action recognition from raw depth image-sequences, provided by an RGB-D camera. The proposal is based on a 3D fully convolutional neural network, named 3DFCNN, which automatically encodes spatio-temporal patterns from depth sequences without %any costly pre-processing. Furthermore, t…
TCL AI Lab（TCL人工智能实验室）；University of Cambridge（剑桥大学）；Samsung AI Cambridge（三星人工智能剑桥研究院）
The ubiquity of camera-enabled mobile devices has lead to large amounts of unlabelled video data being produced at the edge. Although various self-supervised learning (SSL) methods have been proposed to harvest their latent spatio-temporal representations for task-specific training, practical challenges including privacy concerns and communication costs prevent SSL from being deployed at large scales. To mitigate these issues, we propose the use of Federated Learning (FL) to the task of video SSL. In this work, we evaluate the performance of current state-of-the-art (SOTA) video-SSL techniques…
DAP: Doppler-aware Point Network for Heterogeneous mmWave Action Recognition
Peking University（北京大学）；Huazhong University of Science and Technology（华中科技大学）；DJI Technology Company Ltd.（大疆技术创新有限公司）；Christian-Albrechts-Universität zu Kiel（基尔大学）
Neuromorphic vision-based sensors are gaining popularity in recent years with their ability to capture Spatio-temporal events with low power sensing. These sensors record events or spikes over traditional cameras which helps in preserving the privacy of the subject being recorded. These events are captured as per-pixel brightness changes and the output data stream is encoded with time, location, and pixel intensity change information. This paper proposes and benchmarks the performance of fine-tuned conventional vision models on neuromorphic human action recognition and fall detection datasets.…
Federated EndoViT: Pretraining Vision Transformers via Federated Learning on Endoscopic Image Collections
National Center for Tumor Diseases (NCT)（国家肿瘤中心）；Translational Surgical Oncology（转化外科肿瘤学）；Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（医学系和图林根大学技术大学卡尔·古斯塔夫·卡尔医院）；DKFZ, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿大学技术大学医学系和德累斯顿癌症研究中心）；Helmholtz-Zentrum Dresden-Rossendorf (HZDR)（德累斯顿-罗斯多夫亥姆霍兹中心）；Centre for Tactile Internet with Human-in-the-Loop (CeTI), TU Dresden（具有人类在环的触觉互联网中心，德累斯顿技术大学）；Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden（visceral、胸外科和血管外科，医学系和图林根大学技术大学卡尔·古斯塔夫·卡尔医院）；Weldon School of Biomedical Engineering, Purdue University（生物医学工程韦尔登学校，普渡大学）；Medical Oncology, NCT, University Hospital Heidelberg, Germany（医学肿瘤学，国家肿瘤中心，海德堡大学医院，德国）；Medicine I, Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden（医学I，医学系和图林根大学技术大学卡尔·古斯塔夫·卡尔医院）
Indian Institute of Technology Delhi（印度理工学院德里分校）；Indian Institute of Technology Jodhpur（印度理工学院焦特布尔分校）；CSIR-CEERI（印度科学与工业研究理事会-中央电子工程研究所）
Fall detection holds immense importance in the field of healthcare, where timely detection allows for instant medical assistance. In this context, we propose a 3D ConvNet architecture which consists of 3D Inception modules for fall detection. The proposed architecture is a custom version of Inflated 3D (I3D) architecture, that takes compressed measurements of video sequence as spatio-temporal input, obtained from compressive sensing framework, rather than video sequence as input, as in the case of I3D convolutional neural network. This is adopted since privacy raises a huge concern for patient…
Institute of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程研究所）；KITE – Toronto Rehabilitation Institute, University Health Network（大学健康网络KITE-多伦多康复研究所）；Temerty Faculty of Medicine, University of Toronto（多伦多大学特默蒂医学院）；Daphne Cockwell School of Nursing, Ryerson University（瑞尔森大学达芙妮·考克威尔护理学院）
People living with dementia often exhibit behavioural and psychological symptoms of dementia that can put their and others' safety at risk. Existing video surveillance systems in long-term care facilities can be used to monitor such behaviours of risk to alert the staff to prevent potential injuries or death in some cases. However, these behaviours of risk events are heterogeneous and infrequent in comparison to normal events. Moreover, analyzing raw videos can also raise privacy concerns. In this paper, we present two novel privacy-protecting video-based anomaly detection approaches to detect…
Korea University（高丽大学）；Hallym University（翰林大学）；Ajou University（亚洲大学）；UC-Irvine（加州大学尔湾分校）
Autonomous surveillance unmanned aerial vehicles (UAVs) are deployed to observe the streets of the city for any suspicious activities. This paper utilizes surveillance UAVs for the purpose of detecting the presence of a fire in the streets. An extensive database is collected from UAV surveillance drones. With the aid of artificial intelligence (AI), fire stations can swiftly identify the presence of a fire emerging in the neighborhood. Spatio-temporal split learning is applied to this scenario to preserve privacy and globally train a fire classification model. Fires are hazardous natural disas…
Not Only Consistency: Enhance Test-Time Adaptation with Spatio-temporal Inconsistency for Remote Physiological Measurement
Systems Hub, the Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）系统中心）；Information Hub, the Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）信息中心）；School of Information Engineering, Sichuan Agricultural University（四川农业大学信息工程学院）；School of Computing and Information Technology, Great Bay University（大湾大学计算与信息学院）
KITE – Toronto Rehabilitation Institute, University Health Network（KITE – 多伦多康复研究所，大学健康网络）；University of Toronto（多伦多大学）
Fall detection is an important problem from both the health and machine learning perspective. A fall can lead to severe injuries, long term impairments or even death in some cases. In terms of machine learning, it presents a severely class imbalance problem with very few or no training data for falls owing to the fact that falls occur rarely. In this paper, we take an alternate philosophy to detect falls in the absence of their training data, by training the classifier on only the normal activities (that are available in abundance) and identifying a fall as an anomaly. To realize such a classi…
EchoNet-Synthetic: Privacy-preserving Video Generation for Safe Medical Data Sharing
UKRI CDT in AI for Healthcare, Imperial College London（英国帝国理工学院UKRI医疗保健人工智能博士培训中心）；Department of Computing, Imperial College London（帝国理工学院计算系）；Friedrich–Alexander University Erlangen–Nürnberg（埃尔朗根-纽伦堡弗里德里希-亚历山大大学）；School of BMEIS, King’s College London（伦敦国王学院BMEIS学院）；Guy’s and St Thomas’ NHS Foundation Trust（盖伊和圣托马斯NHS基金会信托）；Ultromics Ltd（Ultromics有限公司）；John Radcliffe Hospital, Cardiovascular Clinical Research Facility（约翰拉德克利夫医院心血管临床研究中心）
SFDA-rPPG: Source-Free Domain Adaptive Remote Physiological Measurement with Spatio-Temporal Consistency
School of Computing and Information Technology, Great Bay University（大湾区大学 计算与信息技术学院）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学 国家大数据系统计算技术工程实验室）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR), Singapore（新加坡科技研究局高性能计算研究所）
Privacy-Preserving Video Anomaly Detection: A Survey
Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）；DataLab: Data Science and Informatics, University of California, Davis（加州大学戴维斯分校DataLab:数据科学与信息学）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；School of Data Science and Engineering, East China Normal University（华东师范大学数据科学与工程学院）；College of Future Information Technology, Fudan University（复旦大学未来信息科技学院）；Division of Natural and Applied Sciences, Duke Kunshan University（杜克昆山大学自然与应用科学学院）；Department of Electrical and Computer Engineering, The University of British Columbia（不列颠哥伦比亚大学电气与计算机工程系）
Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generation
School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK（英国伯明翰大学工程学院）；School of Computer Science, University of Birmingham, Birmingham, UK（英国伯明翰大学计算机科学学院）；Department of Applied Health Sciences, University of Birmingham, Birmingham, UK（英国伯明翰大学应用健康科学系）
Holistic Semantic Representation for Navigational Trajectory Generation
Zhejiang University（浙江大学）；Big Graph Center, Hangzhou City University（杭州城市大学大数据中心）；State Key Laboratory of Blockchain and Data Security, Zhejiang University（浙江大学区块链与数据安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
Self-supervised Normality Learning and Divergence Vector-guided Model Merging for Zero-shot Congenital Heart Disease Detection in Fetal Ultrasound Videos
University of Oxford（牛津大学）；University of Technology Nuremberg（纽伦堡工业大学）
VidLeaks: Membership Inference Attacks Against Text-to-Video Models
School of Cyber Science and Technology, Shandong University（山东大学网络科学与技术学院）；Eyeline Labs（Eyeline实验室）；State Key Laboratory of Cryptography and Digital Economy Security, Shandong University（山东大学密码与数字经济安全国家重点实验室）；Shandong Key Laboratory of Artificial Intelligence Security, Shandong University（山东省人工智能安全重点实验室）
STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognition
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School（北京大学深圳研究生院通用人工智能国家重点实验室）；University at Buffalo SUNY（纽约州立大学布法罗分校）；Department of Computer and Information Science, University of Pennsylvania（宾夕法尼亚大学计算机与信息科学系）
