arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.13074cs.CV
收藏

用于动作分割的时间段Transformer

Temporal Segment Transformer for Action Segmentation

ShanghaiTech University(上海科技大学);Baidu Inc(百度公司);Shanghai AI Laboratory(上海人工智能实验室);Durham University(杜伦大学)

Zhichao Liu, Leshan Wang, Desen Zhou, Jian Wang, Songyang Zhang, Yang Bai, Errui Ding, Rui Fan

摘要:

从未修剪的视频中识别人类动作是活动理解中的一项重要任务,并且在建模长距离时间关系方面提出了独特的挑战。最近的作品采用了预测和细化策略,将初始预测转换为全局上下文建模的动作片段。然而,生成的段表示通常是嘈杂的,并表现出不准确的段边界,过度分割和其他问题。为了解决这些问题,我们提出了一种基于注意力的方法,我们称之为\textit{temporal segment transformer},联合段关系建模和去噪。其主要思想是使用段和帧表示之间的注意力来对段表示进行降噪,并且还使用段间注意力来捕获段之间的时间相关性。细化的片段表示用于预测动作标签和调整片段边界,并且基于来自片段掩码的投票产生最终动作分割。我们表明,这种新颖的架构达到了最先进的精度流行的50Salads,GTEA和早餐基准。我们还进行了广泛的消融,以证明我们设计的不同组件的有效性。

英文摘要:

Recognizing human actions from untrimmed videos is an important task in activity understanding, and poses unique challenges in modeling long-range temporal relations. Recent works adopt a predict-and-refine strategy which converts an initial prediction to action segments for global context modeling. However, the generated segment representations are often noisy and exhibit inaccurate segment boundaries, over-segmentation and other problems. To deal with these issues, we propose an attention based approach which we call \textit{temporal segment transformer}, for joint segment relation modeling and denoising. The main idea is to denoise segment representations using attention between segment and frame representations, and also use inter-segment attention to capture temporal correlations between segments. The refined segment representations are used to predict action labels and adjust segment boundaries, and a final action segmentation is produced based on voting from segment masks. We show that this novel architecture achieves state-of-the-art accuracy on the popular 50Salads, GTEA and Breakfast benchmarks. We also conduct extensive ablations to demonstrate the effectiveness of different components of our design.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑