arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06765cs.SD
收藏

从长音频中进行层次化活动识别与描述

Hierarchical Activity Recognition and Captioning from Long-Form Audio

Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, U.K.(视觉、语音和信号处理中心(CVSSP),英国萨里大学)

Peng Zhang, Qingyu Luo, Philip J. B. Jackson, Wenwu Wang

AI总结:

本文提出MultiAct数据集和统一层次模型,用于从长音频中识别复杂活动并生成多分辨率描述。

摘要:

现实世界中的复杂活动在长时间内展开,并表现出层次结构,但大多数先前的工作集中在短片段和孤立事件上。为弥合这一差距,我们引入了MultiAct,一个新的数据集和基准,用于从长音频中进行多级结构化理解人类活动。MultiAct包含长时间的厨房录音,标注在三个语义层次(活动、子活动和事件)上,并配以细粒度描述和高层摘要。我们进一步提出一个统一的层次模型,能够联合执行分类、检测、序列预测和多分辨率描述。在MultiAct上的实验建立了强大的基线,并揭示了在长音频中建模层次和组合结构的关键挑战。未来工作的一个有前途的方向是探索更适合捕捉长音频中复杂、长距离关系的方法。

英文摘要:

Complex activities in real-world audio unfold over extended durations and exhibit hierarchical structure, yet most prior work focuses on short clips and isolated events. To bridge this gap, we introduce MultiAct, a new dataset and benchmark for multi-level structured understanding of human activities from long-form audio. MultiAct comprises long-duration kitchen recordings annotated at three semantic levels (activities, sub-activities and events) and paired with fine-grained captions and high-level summaries. We further propose a unified hierarchical model that jointly performs classification, detection, sequence prediction and multi-resolution captioning. Experiments on MultiAct establish strong baselines and reveal key challenges in modelling hierarchical and compositional structure of long-form audio. A promising direction for future work is the exploration of methods better suited to capturing the complex, long-range relationships in long-form audio.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑