arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.01125cs.CLcs.LG
收藏

多模态事件序列的长程建模与处理

Long-range Modeling and Processing of Multimodal Event Sequences

Jichu Li, Yilun Zhong, Zhiting Li, Feng Zhou, Quyu Kong

AI总结:

本文提出了一种基于LLM的多模态时间点过程框架,通过自适应序列压缩解决长上下文问题,提升多模态事件序列的建模与生成能力。

摘要:

时间点过程(TPPs)已逐渐成为建模异步事件序列的强大工具。尽管最近的进展已扩展TPPs以处理文本信息,但现有方法在生成丰富多模态内容和理解事件动态方面存在局限。关键挑战在于将多模态数据纳入序列会大幅增加序列长度,阻碍基于注意力的模型生成连贯的长文本描述,这些描述需要长距离理解。在本文中,我们提出了一种新的框架,将基于LLM的TPPs扩展到视觉模态,将文本生成作为核心能力与时间和类型预测并列。我们的方法通过基于时间相似性的自适应序列压缩机制解决长上下文问题,该机制在减少序列长度的同时保留了关键模式。我们采用预训练压缩序列后,再通过监督微调进行下游任务的两阶段范式。广泛的实验,包括在具有挑战性的DanmakuTPP-QA基准上的测试,表明我们的方法在预测准确性和生成的文本分析质量方面均优于最先进的基线。

英文摘要:

Temporal point processes (TPPs) have emerged as powerful tools for modeling asynchronous event sequences. While recent advances have extended TPPs to handle textual information, existing approaches are limited in their ability to generate rich, multimodal content and reason about event dynamics. A key challenge is that incorporating multimodal data dramatically increases sequence length, hindering the ability of attention-based models to generate coherent, long-form textual descriptions that require long-range understanding. In this paper, we propose a novel framework that extends LLM-based TPPs to the visual modality, positioning text generation as a core capability alongside time and type prediction. Our approach addresses the long-context problem through an adaptive sequence compression mechanism based on temporal similarity, which reduces sequence length while preserving essential patterns. We employ a two-stage paradigm of pre-training on compressed sequences followed by supervised fine-tuning for downstream tasks. Extensive experiments, including on the challenging DanmakuTPP-QA benchmark, demonstrate that our method outperforms state-of-the-art baselines in both predictive accuracy and the quality of its generated textual analyses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑