EditVerse：通过上下文学习统一图像和视频编辑与生成
EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning
AI总结：
本文提出EditVerse框架，通过统一token序列实现图像和视频生成编辑，利用自注意力机制实现跨模态知识迁移，实验表明其在视频编辑任务中达到SOTA水平。
摘要：
近期基础模型的进展凸显出统一和扩展的趋势，展示了跨领域涌现能力。尽管图像生成和编辑已从专用任务转向统一框架，但视频生成和编辑仍因架构限制和数据稀缺而碎片化。本文引入EditVerse，一种统一的图像和视频生成与编辑框架。通过将所有模态（文本、图像和视频）表示为统一的token序列，EditVerse利用自注意力机制实现稳健的上下文学习、自然的跨模态知识迁移以及灵活处理任意分辨率和持续时间的输入输出。为解决视频编辑训练数据不足的问题，我们设计了可扩展的数据管道，收集了232,000个视频编辑样本，并将其与大规模图像和视频数据集结合进行联合训练。此外，我们提出了EditVerseBench，首个针对指令式视频编辑的基准测试，涵盖多样化的任务和分辨率。广泛实验和用户研究显示，EditVerse在视频编辑任务中达到SOTA水平，超越现有开源和商业模型，同时在跨模态编辑和生成能力上表现出涌现能力。
英文摘要：
Recent advances in foundation models highlight a clear trend toward unification and scaling, showing emergent capabilities across diverse domains. While image generation and editing have rapidly transitioned from task-specific to unified frameworks, video generation and editing remain fragmented due to architectural limitations and data scarcity. In this work, we introduce EditVerse, a unified framework for image and video generation and editing within a single model. By representing all modalities, i.e., text, image, and video, as a unified token sequence, EditVerse leverages self-attention to achieve robust in-context learning, natural cross-modal knowledge transfer, and flexible handling of inputs and outputs with arbitrary resolutions and durations. To address the lack of video editing training data, we design a scalable data pipeline that curates 232K video editing samples and combines them with large-scale image and video datasets for joint training. Furthermore, we present EditVerseBench, the first benchmark for instruction-based video editing covering diverse tasks and resolutions. Extensive experiments and user studies demonstrate that EditVerse achieves state-of-the-art performance, surpassing existing open-source and commercial models, while exhibiting emergent editing and generation abilities across modalities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
