We present a self-supervised framework that learns population-level codes for arbitrary ensembles of neural recordings at scale. We address key challenges in scaling models with neural time-series data, namely, sparse and variable electrode distribution across subjects and datasets. The Population Transformer (PopT) stacks on top of pretrained temporal embeddings and enhances downstream decoding by enabling learned aggregation of multiple spatially-sparse data channels. The pretrained PopT lowers the amount of data required for downstream decoding experiments, while increasing accuracy, even on held-out subjects and tasks. Compared to end-to-end methods, this approach is computationally lightweight, while achieving similar or better decoding performance. We further show how our framework is generalizable to multiple time-series embeddings and neural data modalities. Beyond decoding, we interpret the pretrained and fine-tuned PopT models to show how they can be used to extract neuroscience insights from large amounts of data. We release our code as well as a pretrained PopT to enable off-the-shelf improvements in multi-channel intracranial data decoding and interpretability. Code is available at https://github.com/czlwang/PopulationTransformer.
Modeling neural population dynamics underlying noisy single-trial spiking activities is essential for relating neural observation and behavior. A recent non-recurrent method - Neural Data Transformers (NDT) - has shown great success in capturing neural dynamics with low inference latency without an explicit dynamical model. However, NDT focuses on modeling the temporal evolution of the population activity while neglecting the rich covariation between individual neurons. In this paper we introduce SpatioTemporal Neural Data Transformer (STNDT), an NDT-based architecture that explicitly models r…
Princeton University（普林斯顿大学）；Columbia University（哥伦比亚大学）；University of Pittsburgh（匹兹堡大学）；Massachusetts Institute of Technology（麻省理工学院）；Neuralink Corp.（Neuralink公司）；Stanford University（斯坦福大学）；University of Chicago（芝加哥大学）；Northwestern University（西北大学）；Stony Brook University（石溪大学）；Emory University（埃默里大学）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）
Advances in neural recording present increasing opportunities to study neural activity in unprecedented detail. Latent variable models (LVMs) are promising tools for analyzing this rich activity across diverse neural systems and behaviors, as LVMs do not depend on known relationships between the activity and external experimental variables. However, progress with LVMs for neuronal population activity is currently impeded by a lack of standardization, resulting in methods being developed and compared in an ad hoc manner. To coordinate these modeling efforts, we introduce a benchmark suite for l…
Grounded situation recognition is the task of predicting the main activity, entities playing certain roles within the activity, and bounding-box groundings of the entities in the given image. To effectively deal with this challenging task, we introduce a novel approach where the two processes for activity classification and entity estimation are interactive and complementary. To implement this idea, we propose Collaborative Glance-Gaze TransFormer (CoFormer) that consists of two modules: Glance transformer for activity classification and Gaze transformer for entity estimation. Glance transform…
Data augmentation is a widely used technique in classification to increase data used in training. It improves generalization and reduces amount of annotated human activity data needed for training which reduces labour and time needed with the dataset. Sensor time-series data, unlike images, cannot be augmented by computationally simple transformation algorithms. State of the art models like Recurrent Generative Adversarial Networks (RGAN) are used to generate realistic synthetic data. In this paper, transformer based generative adversarial networks which have global attention on data, are comp…
A Gravity-informed Spatiotemporal Transformer for Human Activity Intensity Prediction
National Engineering Research Center of Geographic Information System, China University of Geosciences (Wuhan)（中国地质大学（武汉）国家地理信息系统工程技术研究中心）；JD iCity, JD Technology（京东科技京东智能城市）
University of Edinburgh（爱丁堡大学）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
Neural population activity relating to behaviour is assumed to be inherently low-dimensional despite the observed high dimensionality of data recorded using multi-electrode arrays. Therefore, predicting behaviour from neural population recordings has been shown to be most effective when using latent variable models. Over time however, the activity of single neurons can drift, and different neurons will be recorded due to movement of implanted neural probes. This means that a decoder trained to predict behaviour on one day performs worse when tested on a different day. On the other hand, eviden…
Providing care for ageing populations is an onerous task, and as life expectancy estimates continue to rise, the number of people that require senior care is growing rapidly. This paper proposes a methodology based on Transformer Neural Networks to classify the activities of a resident within an ambient sensor based environment. We also propose a methodology to pre-train Transformers in a self-supervised manner, as a hybrid autoencoder-classifier model instead of using contrastive loss. The social impact of the research is considered with wider benefits of the approach and next steps for ident…
An Innovative Next Activity Prediction Using Process Entropy and Dynamic Attribute-Wise-Transformer in Predictive Business Process Monitoring
Department of Electrical and Computer Engineering, Faculty of Engineering and Computer Science, University of Victoria（维多利亚大学工程与计算机科学学院电气与计算机工程系）；Department of Mechanical Engineering, Faculty of Engineering and Computer Science, University of Victoria（维多利亚大学工程与计算机科学学院机械工程系）
The pervasiveness of Wi-Fi signals provides significant opportunities for human sensing and activity recognition in fields such as healthcare. The sensors most commonly used for passive Wi-Fi sensing are based on passive Wi-Fi radar (PWR) and channel state information (CSI) data, however current systems do not effectively exploit the information acquired through multiple sensors to recognise the different activities. In this paper, we explore new properties of the Transformer architecture for multimodal sensor fusion. We study different signal processing techniques to extract multiple image-ba…
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Technische Universität Kaiserslautern（凯泽斯劳滕工业大学）
Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human-computer interaction applications. However, they are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, they still face significant challenging problems with performance degradation due to large ga…
Several popular Transformer based language models have been found to be successful for text-driven brain encoding. However, existing literature leverages only pretrained text Transformer models and has not explored the efficacy of task-specific learned Transformer representations. In this work, we explore transfer learning from representations learned for ten popular natural language processing tasks (two syntactic and eight semantic) for predicting brain responses from two diverse datasets: Pereira (subjects reading sentences from paragraphs) and Narratives (subjects listening to the spoken s…
As a fundamental problem in ubiquitous computing and machine learning, sensor-based human activity recognition (HAR) has drawn extensive attention and made great progress in recent years. HAR aims to recognize human activities based on the availability of rich time-series data collected from multi-modal sensors such as accelerometers and gyroscopes. However, recent deep learning methods are focusing on one view of the data, i.e., the temporal view, while shallow methods tend to utilize the hand-craft features for recognition, e.g., the statistics view. In this paper, to extract a better featur…
Self-Clustering Graph Transformer Approach to Model Resting-State Functional Brain Activity
Georgia State University（佐治亚州立大学）；Tri-Institutional Center for Translational Research in Neuroimaging and Data Science（三方神经成像与数据科学转化研究中心）；Georgia Institute of Technology（佐治亚理工学院）
ConViViT -- A Deep Neural Network Combining Convolutions and Factorized Self-Attention for Human Activity Recognition
Efrei Research Lab（Efrei研究实验室）；Paris Pantheon-Assas University（巴黎先贤祠-阿萨斯大学）；Paris Panthéon-Assas University（巴黎先贤祠-阿萨斯大学）；ICube UMR 7357, CNRS（法国国家科学研究中心ICube UMR 7357实验室）；Université de Strasbourg（斯特拉斯堡大学）；Caplogy SAS（Caplogy公司）
In this work, we develop a privacy-by-design generative model for synthesizing the activity diary of the travel population using state-of-art deep learning approaches. This proposed approach extends literature on population synthesis by contributing novel deep learning to the development and application of synthetic travel data while guaranteeing privacy protection for members of the sample population on which the synthetic populations are based. First, we show a complete de-generalization of activity diaries to simulate the socioeconomic features and longitudinal sequences of geographically a…
BLEND: Behavior-guided Neural Population Dynamics Modeling via Privileged Knowledge Distillation
The Hong Kong University of Science and Technology（香港科技大学）；Beijing Institute of Collaborative Innovation（北京协同创新研究院）；Peking University（北京大学）；Beihang University（北京航空航天大学）
We introduce a novel machine learning approach to leverage historical and contemporary maps and systematically predict economic statistics. Our simple algorithm extracts meaningful features from the maps based on their color compositions for predictions. We apply our method to grid-level population levels in Sub-Saharan Africa in the 1950s and South Korea in 1930, 1970, and 2015. Our results show that maps can reliably predict population density in the mid-20th century Sub-Saharan Africa using 9,886 map grids (5km by 5 km). Similarly, contemporary South Korean maps can generate robust predicti…
ALERT Open Dataset and Input-Size-Agnostic Vision Transformer for Driver Activity Recognition using IR-UWB
Department of ICE, Hanbat National University, Daejeon, Republic of Korea（信息电子工程系，翰 bam国立大学）；Department of ECE and INMC, Seoul National University, Seoul, Republic of Korea（电子工程系和智能纳米材料中心，首尔国立大学）；Department of ECE and INMC, Seoul National University（电子工程系和智能纳米材料中心，首尔国立大学）
Are Anxiety Detection Models Generalizable? A Cross-Activity and Cross-Population Study Using Wearables
Indian Institute of Science Education and Research Bhopal (IISERB)（印度科学教育与研究学院博帕尔分校（IISERB））；All India Institute of Medical Sciences Bhopal（全印医学科学学院博帕尔分校）
In the recent years there has been a growing interest in techniques able to automatically recognize activities performed by people. This field is known as Human Activity recognition (HAR). HAR can be crucial in monitoring the wellbeing of the people, with special regard to the elder population and those people affected by degenerative conditions. One of the main challenges concerns the diversity of the population and how the same activities can be performed in different ways due to physical characteristics and life-style. In this paper we explore the possibility of exploiting physical characte…
The adoption of electronic health records (EHR) has become universal during the past decade, which has afforded in-depth data-based research. By learning from the large amount of healthcare data, various data-driven models have been built to predict future events for different medical tasks, such as auto diagnosis and heart-attack prediction. Although EHR is abundant, the population that satisfies specific criteria for learning population-specific tasks is scarce, making it challenging to train data-hungry deep learning models. This study presents the Claim Pre-Training (Claim-PT) framework, a…
