MTGA：面向事件唇语识别的多视角时间粒度对齐聚合
MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading
AI总结：
针对事件唇语识别中事件帧聚合导致细粒度时间信息丢失的问题，提出MTGA框架，通过时间分段体素图列表、时间粒度对齐的时空融合模块及带位置编码的时间聚合模块，有效提升识别性能并优于现有事件和视频方法。
摘要：
唇语识别旨在利用说话者唇部运动的视觉信息来识别词语和句子。现有基于事件的唇语识别方案通过整合不同帧率分支来学习不同粒度的时空特征。然而，将事件聚合成事件帧不可避免地会导致帧内细粒度时间信息的丢失。为弥补这一缺陷，我们提出了一种新颖的框架，称为多视角时间粒度对齐聚合（MTGA）。具体而言，我们首先提出一种新颖的事件表示方法，即时间分段体素图列表，其中最重要的局部体素在时间上被连接成一个图列表。然后，我们设计了一个基于时间粒度对齐的时空融合模块，将从事件帧中提取的全局空间特征与体素图列表中包含的局部相对空间和时间特征进行有效对齐与整合。最后，我们设计了一个融合位置编码的时间聚合模块，从而能够捕获局部绝对空间信息和全局时间信息。实验表明，我们的方法优于基于事件和基于视频的唇语识别同类方法。
英文摘要：
Lip-reading is to utilize the visual information of the speaker's lip movements to recognize words and sentences. Existing event-based lip-reading solutions integrate different frame rate branches to learn spatio-temporal features of varying granularities. However, aggregating events into event frames inevitably leads to the loss of fine-grained temporal information within frames. To remedy this drawback, we propose a novel framework termed Multi-view Temporal Granularity aligned Aggregation (MTGA). Specifically, we first present a novel event representation method, namely time-segmented voxel graph list, where the most significant local voxels are temporally connected into a graph list. Then we design a spatio-temporal fusion module based on temporal granularity alignment, where the global spatial features extracted from event frames, together with the local relative spatial and temporal features contained in voxel graph list are effectively aligned and integrated. Finally, we design a temporal aggregation module that incorporates positional encoding, which enables the capture of local absolute spatial and global temporal information. Experiments demonstrate that our method outperforms both the event-based and video-based lip-reading counterparts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
