Developing useful interfaces between brains and machines is a grand challenge of neuroengineering. An effective interface has the capacity to not only interpret neural signals, but predict the intentions of the human to perform an action in the near future; prediction is made even more challenging outside well-controlled laboratory experiments. This paper describes our approach to detect and to predict natural human arm movements in the future, a key challenge in brain computer interfacing that has never before been attempted. We introduce the novel Annotated Joints in Long-term ECoG (AJILE) dataset; AJILE includes automatically annotated poses of 7 upper body joints for four human subjects over 670 total hours (more than 72 million frames), along with the corresponding simultaneously acquired intracranial neural recordings. The size and scope of AJILE greatly exceeds all previous datasets with movements and electrocorticography (ECoG), making it possible to take a deep learning approach to movement prediction. We propose a multimodal model that combines deep convolutional neural networks (CNN) with long short-term memory (LSTM) blocks, leveraging both ECoG and video modalities. We demonstrate that our models are able to detect movements and predict future movements up to 800 msec before movement initiation. Further, our multimodal movement prediction models exhibit resilience to simulated ablation of input neural signals. We believe a multimodal approach to natural neural decoding that takes context into account is critical in advancing bioelectronic technologies and human neuroscience.
Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings
University of Washington（华盛顿大学）；Center for Neurotechnology（神经技术中心）；University of Washington Institute for Neuroengineering（华盛顿大学神经工程研究所）；eScience Institute, University of Washington（华盛顿大学电子科学研究所）
Robust Tracking and Behavioral Modeling of Movements of Biological Collectives from Ordinary Video Recordings
Binghamton University, State University of New York（纽约州立大学宾汉姆顿分校）；Waseda University（早稻田大学）；Drexel University（德雷塞尔大学）；SUNY College of Environmental Science and Forestry（纽约州立大学环境科学与林业学院）
Towards Scalable Newborn Screening: Automated General Movement Assessment in Uncontrolled Settings
ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）；University of Regensburg（莱茵河畔埃尔兰根大学）；University Children’s Hospital（儿童医院）；University Children’s Hospital Regensburg（埃尔兰根儿童医院）
Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements
Biomedical Engineering Department, Northwestern University（西北大学生物医学工程系）；Shirley Ryan AbilityLab（谢里尔·瑞安能力实验室）；Department of Organismal Biology and Anatomy, University of Chicago（芝加哥大学器官生物学与解剖学系）；Department of Physical Medicine and Rehabilitation, Northwestern University（西北大学物理医学与康复系）
Open video data sharing in developmental and behavioural science
University Medical Center Göttingen（哥廷根大学医学中心）；Karolinska Institutet（卡罗林斯卡学院）；Medical University of Graz（格拉茨医科大学）；Leibniz-ScienceCampus Primate Cognition（莱布尼茨灵长类认知科学园区）；David Geffen UCLA School of Medicine（加州大学洛杉矶分校大卫·格芬医学院）；Region Stockholm（斯德哥尔摩地区）；Curtin University（科廷大学）；Victoria University of Wellington（惠灵顿维多利亚大学）
Multimodal video and IMU kinematic dataset on daily life activities using affordable devices (VIDIMU)
University of Valladolid（巴利亚多利德大学）；University of Applied Sciences and Arts Western Switzerland (HES-SO) Valais-Wallis（瑞士西部应用科学与艺术大学 (HES-SO) 瓦莱-沃利斯分校）；University of Geneva（日内瓦大学）
Automated Classification of General Movements in Infants Using a Two-stream Spatiotemporal Fusion Network
Hiroshima University（广岛大学）；Prefectural University of Hiroshima（广岛县立大学）；University of Genoa（热那亚大学）；Istituto G.Gaslini Pediatric Hospital（G.加斯利尼儿科研究所）；Italian Institute of Technology（意大利技术研究院）
BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis
University of Edinburgh（爱丁堡大学）；Nanyang Technological University（南洋理工大学）；Shanghai AI Laboratory（上海人工智能实验室）
音频调节舞蹈动作合成的生成模型将音乐特征映射到舞蹈动作。模型被训练成将运动模式与音频模式相关联，通常没有人体的明确知识。这种方法依赖于一些假设：强烈的音乐舞蹈相关性，受控的运动数据和相对简单的姿势和动作。这些特点都发现在所有现有的数据集的舞蹈动作合成，确实最近的方法可以取得很好的效果。我们介绍了一个新的数据集，旨在挑战这些常见的假设，编译一组动态的舞蹈序列显示复杂的人体姿势。我们专注于霹雳舞的特点杂技动作和纠结的姿势。我们的数据来自Red Bull BC One比赛视频。由于舞蹈的复杂性以及多个移动摄像机的记录设置，从这些视频中估计人类关键点是困难的。我们采用混合标记管道，利用深度估计模型以及手动注释，以降低成本获得高质量的关键点序列。我们的努力产生了BRACE数据集，其中包含超过3小时30分钟的密集注释姿势。我们在BRACE上测试了最先进的方法，显示了它们在复杂序列上评估时的局限性。我们的数据集可以很容易地促进舞蹈动作合成的进步。随着复杂的姿势和快速的动作，模型被迫超越学习模式之间的映射，更有效地了解身体结构和动作。
Quantitative Movement Testing: Measuring Chronic Pain Patient Movements from a Single Smartphone Video
Nuffield Department of Clinical Neurosciences, University of Oxford（临床神经科学系,Nuffield大学,牛津大学）；Max Planck Institute of Biological Cybernetics（生物信息学研究所）；Oxford Gait Laboratory, University of Oxford（牛津大学步态实验室）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Institute of Biomedical Engineering, University of Oxford（生物医学工程研究所,牛津大学）；Mayo Clinic（梅奥诊所）
A store-and-forward cloud-based telemonitoring system for automatic assessing dysarthria evolution in neurological diseases from video-recording analysis
AIDAPT S.r.l（艾达普特有限公司）；Centro Clinico NeuroMuscular Omnicentre (NeMO), Fondazione Serena Onlus（尼莫神经肌肉临床中心（尼莫中心），塞雷娜公益基金会）；Azienda Ospedaliero-Universitaria delle Marche（马尔凯地区大学医院）；Università Degli Studi di Macerata（马切拉塔大学）；Scuola Superiore Sant’Anna（圣安娜高等学校）
本文提出一种基于云存储转发的远程监测系统，集成面部标志 Mask RCNN 卷积神经网络，通过视频分析自动定位面部标志并评估神经系统疾病患者的构音障碍演变，在 Toronto NeuroFace 数据集和真实 ALS 患者中验证了有效性。
A Spatial-Temporal Transformer based Framework For Human Pose Assessment And Correction in Education Scenarios
Fudan-Xiding AI Joint LAB（复旦-西鼎人工智能联合实验室）；Shangshai Xiding AI Research Center Co., Ltd（上海西鼎人工智能研究中心有限公司）；R&D Center Shangshai Xiding AI Research Center Co., Ltd（上海西鼎人工智能研究中心有限公司研发中心）；School of Information Science and Technology Fudan University（复旦大学信息科学与技术学院）
Sport Task: Fine Grained Action Detection and Classification of Table Tennis Strokes from Videos for MediaEval 2022
Max Planck Institute for Evolutionary Anthropology（马克斯·普朗克进化人类学研究所）；La Rochelle University（拉罗谢尔大学）；University of Bordeaux（波尔多大学）；CNRS（法国国家科学研究中心）；Bordeaux INP（波尔多国立理工学院）
