HyperTokens: 通过控制令牌动态实现连续视频-语言理解
DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding
AI总结：
本文提出HyperTokens，通过按需生成微调令牌，控制提示更新并保持内存固定，通过元启发式正则化抑制遗忘，提升连续视频问答任务的准确率与保留率。
摘要：
持续视频问答任务受任务间干扰和存储特定提示成本高所限。我们引入HyperTokens，一种基于transformer的令牌生成器，可按需生成微调令牌，提供对提示更新的显式控制，同时保持内存固定。为抑制遗忘，我们提出元启发式正则化器，通过前瞻性避免任务特定的尖锐方向，并将演进生成器锚定于先前任务。我们进一步将目标与尖锐度感知优化联系起来，提供洞察为何它鼓励更平坦的跨任务最小值并提高保留率。除正则化外，HyperTokens利用轻量级辅助多模态监督通过共享生成权重；受因果视角引导，我们设计可行目标和替代互信息损失以正则化反因果跨模态方向。在两个标准持续视频问答基准上，HyperTokens在显著降低遗忘的同时实现了更高的平均准确率。最后，我们引入一个具有挑战性的跨模态ImageQA->VideoQA协议，并展示HyperTokens在此设置中实现了稳健的持续迁移。
英文摘要：
Continual VideoQA with multimodal LLMs remains challenging because sequential adaptation induces task interference, while storing task-specific prompts becomes impractical as task sequences grow. We introduce DynaTokens, a transformer-based token generator that dynamically produces fine-tuning tokens on demand, enabling task-adaptive prompt updates through shared generation weights. To mitigate forgetting, we introduce meta-learning-inspired regularisers that look ahead to avoid task-specific sharp update directions while anchoring the evolving generator to prior-task behaviours. We theoretically connect this objective to sharpness-aware optimisation, showing how it favours flatter cross-task minima and improves retention. DynaTokens combines gradient-free routing based on robust pretrained token and visual embeddings with lightweight auxiliary multimodal supervision, reducing router drift during continual adaptation. Across standard continual VideoQA benchmarks, DynaTokens achieves higher average accuracy and substantially lower forgetting than strong baselines. It also improves zero-shot generalisation and remains effective in longer domain-incremental sequences with extended task shifts. Finally, we introduce a challenging ImageQA->VideoQA protocol and show that DynaTokens enables robust cross-modal continual transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
