arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.08476cs.CVcs.AI
收藏

RMem:受限内存库改进视频对象分割

RMem: Restricted Memory Banks Improve Video Object Segmentation

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)

Junbao Zhou, Ziqi Pang, Yu-Xiong Wang

AI总结:

针对视频对象分割任务,提出受限内存库策略RMem,通过限制内存库规模减少冗余干扰、平衡帧信息、缩小训练推理差异,引入时序位置嵌入,在挑战性场景及相关数据集上取得新SOTA。

摘要:

随着近期视频对象分割(VOS)基准向更具挑战性的场景演进,我们重新审视了一种简单却被忽视的策略:限制内存库的大小。这与当前普遍采用的扩大内存库以容纳大量历史信息的做法截然不同。我们专门设计的“内存解码”研究为该策略提供了关键的支撑性洞见:扩大内存库看似有益,实则会因冗余信息造成的混淆,增加VOS模块解码相关特征的难度。通过将内存库限制在有限数量的关键帧内,我们实现了VOS准确率的显著提升。该过程平衡了帧的重要性与新鲜度,从而在有限容量内维持具备信息价值的内存库。此外,与持续扩张的内存库相比,受限内存库缩小了训练与推理在内存长度上的差异。这为时序推理带来了新的可能,也让我们得以引入此前被忽视的“时序位置嵌入”。最终,我们的洞见体现在“RMem”(“R”代表受限)中——这是一种简单却有效的VOS改进方案,在挑战性VOS场景中表现优异,并在对象状态变化(VOST数据集)和长视频(Long Videos数据集)任务上创下新的最优水平。我们的代码和演示可在https://restricted-memory.github.io/获取。

英文摘要:

With recent video object segmentation (VOS) benchmarks evolving to challenging scenarios, we revisit a simple but overlooked strategy: restricting the size of memory banks. This diverges from the prevalent practice of expanding memory banks to accommodate extensive historical information. Our specially designed "memory deciphering" study offers a pivotal insight underpinning such a strategy: expanding memory banks, while seemingly beneficial, actually increases the difficulty for VOS modules to decode relevant features due to the confusion from redundant information. By restricting memory banks to a limited number of essential frames, we achieve a notable improvement in VOS accuracy. This process balances the importance and freshness of frames to maintain an informative memory bank within a bounded capacity. Additionally, restricted memory banks reduce the training-inference discrepancy in memory lengths compared with continuous expansion. This fosters new opportunities in temporal reasoning and enables us to introduce the previously overlooked "temporal positional embedding." Finally, our insights are embodied in "RMem" ("R" for restricted), a simple yet effective VOS modification that excels at challenging VOS scenarios and establishes new state of the art for object state changes (on the VOST dataset) and long videos (on the Long Videos dataset). Our code and demo are available at https://restricted-memory.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑