arXiv 2511.114342025-11-14cs.CV

WEAVE：上下文交错式理解与生成能力的释放与基准评测

WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation

National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；University of Maryland, College Park（马里兰大学学院公园分校）；Zhejiang University（浙江大学）

Wei Chow, Jiachun Pan, Yongyuan Liang, Mingze Zhou, Xue Song, Liyu Jia, Saining Zhang, Siliang Tang, Juncheng Li, Fengda Zhang, Weijia Wu, Hanwang Zhang, Tat-Seng Chua

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有多模态基准缺乏多轮上下文交互评测的问题，提出WEAVE数据集与基准套件，验证其训练可提升模型多模态能力并揭示现有方法局限，为相关研究提供基础。

摘要：

统一多模态模型（UMMs）的近期进展在视觉理解与生成领域取得了瞩目成果，但现有数据集与基准主要聚焦单轮交互，无法捕捉现实世界图像创作与编辑的多轮、上下文依赖特性。为填补这一空白，我们提出WEAVE，这是首个面向上下文交错跨模态理解与生成的套件，包含两个互补部分：WEAVE-100k是包含10万份交错样本的大规模数据集，覆盖超37万轮对话与50万张图像，涵盖需要基于历史上下文推理的理解、编辑、生成任务；WEAVEBench是基于480张图像构建、包含100项任务的人工标注基准，采用结合参考图像、原始图像与编辑指令组合的混合VLM评判器评估框架，可评测模型在多领域的多轮生成、视觉记忆、世界知识推理能力。实验表明，在WEAVE-100k上训练可提升模型的视觉理解、图像编辑、理解-生成协同能力，还能促使UMMs涌现出视觉记忆能力；同时在WEAVEBench上的大量评测暴露了当前方法在多轮、上下文感知的图像生成与编辑上仍存在持续局限与挑战。我们认为WEAVE为多模态领域研究上下文交错式理解与生成提供了视角与基础。

英文摘要：

Recent advances in unified multimodal models (UMMs) have enabled impressive progress in visual comprehension and generation. However, existing datasets and benchmarks focus primarily on single-turn interactions, failing to capture the multi-turn, context-dependent nature of real-world image creation and editing. To address this gap, we present WEAVE, the first suite for in-context interleaved cross-modality comprehension and generation. Our suite consists of two complementary parts. WEAVE-100k is a large-scale dataset of 100K interleaved samples spanning over 370K dialogue turns and 500K images, covering comprehension, editing, and generation tasks that require reasoning over historical context. WEAVEBench is a human-annotated benchmark with 100 tasks based on 480 images, featuring a hybrid VLM judger evaluation framework based on both the reference image and the combination of the original image with editing instructions that assesses models' abilities in multi-turn generation, visual memory, and world-knowledge reasoning across diverse domains. Experiments demonstrate that training on WEAVE-100k enables vision comprehension, image editing, and comprehension-generation collaboration capabilities. Furthermore, it facilitates UMMs to develop emergent visual-memory capabilities, while extensive evaluations on WEAVEBench expose the persistent limitations and challenges of current approaches in multi-turn, context-aware image generation and editing. We believe WEAVE provides a view and foundation for studying in-context interleaved comprehension and generation for multi-modal community.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。