arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.07034cs.CV

一种用于基于事件的人体姿态估计的时序密集连接循环网络

A Temporal Densely Connected Recurrent Network for Event-based Human Pose Estimation

Hunan Normal University(湖南师范大学);Zhejiang University of Technology(浙江工业大学);City University of Hong Kong(香港城市大学);Soochow University(苏州大学)

Zhanpeng Shao, Wen Zhou, Wuzhen Wang, Jianyu Yang, Youfu Li

AI总结:

本文提出一种密集连接循环架构,通过建模时序与非时序几何一致性来恢复基于事件相机的人体姿态估计中缺失的身体部位,并构建了大规模多模态数据集验证其有效性。

摘要:

Event camera是一种新兴的仿生视觉传感器,能够异步报告每个像素的亮度变化。它具有高动态范围、高速响应和低功耗的显著优势,使其能够在不受控的环境中最好地捕捉局部运动。这促使我们挖掘event camera在人体姿态估计中的潜力,因为基于event camera的人体姿态估计鲜有探索。然而,由于从传统基于帧的相机到event camera的范式转变,一个时间间隔内的事件信号包含的信息非常有限,因为event camera只能捕捉运动的身体部位而忽略静态的身体部位,导致某些部位在该时间间隔内不完整甚至消失。本文提出了一种新颖的密集连接循环架构来解决信息不完整的问题。通过这种循环架构,我们不仅能显式建模时序的几何一致性,还能建模时间步之间非时序的几何一致性,从而从先前的帧中累积信息以恢复整个人体,实现从事件数据中稳定且准确的人体姿态估计。此外,为了更好地评估我们的模型,我们收集了一个大规模的多模态基于事件的数据集,并附带人体姿态标注,据我们所知,这是迄今为止最具挑战性的数据集。在两个公开数据集和我们自己的数据集上的实验结果证明了我们方法的有效性和优势。代码可在线获取以促进未来的研究。

英文摘要:

Event camera is an emerging bio-inspired vision sensors that report per-pixel brightness changes asynchronously. It holds noticeable advantage of high dynamic range, high speed response, and low power budget that enable it to best capture local motions in uncontrolled environments. This motivates us to unlock the potential of event cameras for human pose estimation, as the human pose estimation with event cameras is rarely explored. Due to the novel paradigm shift from conventional frame-based cameras, however, event signals in a time interval contain very limited information, as event cameras can only capture the moving body parts and ignores those static body parts, resulting in some parts to be incomplete or even disappeared in the time interval. This paper proposes a novel densely connected recurrent architecture to address the problem of incomplete information. By this recurrent architecture, we can explicitly model not only the sequential but also non-sequential geometric consistency across time steps to accumulate information from previous frames to recover the entire human bodies, achieving a stable and accurate human pose estimation from event data. Moreover, to better evaluate our model, we collect a large scale multimodal event-based dataset that comes with human pose annotations, which is by far the most challenging one to the best of our knowledge. The experimental results on two public datasets and our own dataset demonstrate the effectiveness and strength of our approach. Code can be available online for facilitating the future research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑