arXiv 2506.096502025-10-03cs.CVcs.LGcs.MMcs.ROeess.IV

HopaDIFF：面向多人场景下指代人体动作分割的整体-局部感知傅里叶条件扩散模型

HopaDIFF: Holistic-Partial Aware Fourier Conditioned Diffusion for Referring Human Action Segmentation in Multi-Person Scenarios

Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Beijing Institute of Technology（北京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hunan University（湖南大学）；Shanghai AI Lab（上海人工智能实验室）；HEBUST

Kunyu Peng, Junchao Huang, Xiangsheng Huang, Di Wen, Junwei Zheng, Yufan Chen, Kailun Yang, Jiamin Wu, Chongqing Hao, Rainer Stiefelhagen

备注：: Accepted to NeurIPS 2025. The dataset and code are available at https://github.com/KPeng9510/HopaDIFF

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有动作分割方法难以适配多人场景的问题，该研究构建了首个指代人体动作分割数据集RHAS133，并提出HopaDIFF框架，通过跨输入门注意力xLSTM和傅里叶条件提升性能，取得了当前最优结果。

摘要：

动作分割是高层视频理解中的核心挑战，旨在将未修剪的视频划分为多个片段，并为每个片段分配预定义动作集合中的标签。现有方法主要针对具有固定动作序列的单人活动，忽略了多人场景。在本研究中，我们率先开展多人场景下文本参考引导的人体动作分割研究，即通过文本描述指定待分割的目标人物。我们构建了首个用于指代人体动作分割的数据集RHAS133，该数据集源自133部电影，标注了137种细粒度动作，包含33小时视频数据，并配有对应这一新任务的文本描述。使用基于VLM（视觉语言模型）的特征提取器在RHAS133上对现有动作分割方法进行基准测试，结果显示这些方法性能有限，且对目标人物的视觉线索聚合能力较差。为解决这一问题，我们提出了一种整体-局部感知的傅里叶条件扩散框架HopaDIFF，它利用新型的跨输入门注意力xLSTM增强整体-局部长程推理能力，并通过新型傅里叶条件引入更细粒度的控制，以提升动作分割的生成效果。HopaDIFF在多种评估设置下于RHAS133数据集上取得了最优结果。数据集和代码可在https://github.com/KPeng9510/HopaDIFF 获取。

英文摘要：

Action segmentation is a core challenge in high-level video understanding, aiming to partition untrimmed videos into segments and assign each a label from a predefined action set. Existing methods primarily address single-person activities with fixed action sequences, overlooking multi-person scenarios. In this work, we pioneer textual reference-guided human action segmentation in multi-person settings, where a textual description specifies the target person for segmentation. We introduce the first dataset for Referring Human Action Segmentation, i.e., RHAS133, built from 133 movies and annotated with 137 fine-grained actions with 33h video data, together with textual descriptions for this new task. Benchmarking existing action segmentation methods on RHAS133 using VLM-based feature extractors reveals limited performance and poor aggregation of visual cues for the target person. To address this, we propose a holistic-partial aware Fourier-conditioned diffusion framework, i.e., HopaDIFF, leveraging a novel cross-input gate attentional xLSTM to enhance holistic-partial long-range reasoning and a novel Fourier condition to introduce more fine-grained control to improve the action segmentation generation. HopaDIFF achieves state-of-the-art results on RHAS133 in diverse evaluation settings. The dataset and code are available at https://github.com/KPeng9510/HopaDIFF.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。