arXiv 2507.128412025-10-28cs.CV

AnyCap Project：面向可控全模态字幕生成的统一框架、数据集与基准

AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioning

Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）

Yiming Ren, Zhiqiang Lin, Yu Li, Gao Meng, Weiyun Wang, Junjie Wang, Zicheng Lin, Jifeng Dai, Yujiu Yang, Wenhai Wang, Ruihang Chu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对可控字幕生成模型缺细粒度控制、缺可靠评估与数据的问题，提出AnyCap Project，含即插即用可控框架ACM、30万条多模态数据集ACD及解耦评估的AnyCapEval基准，显著提升多基准上的字幕质量。

摘要：

可控字幕生成对精准多模态对齐与指令遵循至关重要，但现有模型往往缺乏细粒度控制能力与可靠的评估协议。为填补这一空白，我们提出AnyCap Project，这是一套涵盖模型、数据集与评估的一体化解决方案。我们推出AnyCapModel（简称ACM，即任意字幕模型），这是一个轻量级即插即用框架，无需重新训练基础模型，即可增强现有基础模型在全模态字幕生成任务中的可控性。ACM复用基础模型生成的原始字幕，同时结合用户指令与模态特征生成质量更优的字幕。为解决可控多模态字幕生成领域的数据稀缺问题，我们构建了AnyCapDataset（简称ACD，即任意字幕数据集），覆盖三种模态、28种用户指令类型以及30万条高质量数据条目。我们进一步提出AnyCapEval基准，该基准通过解耦内容准确性与风格保真度，为可控字幕生成提供更可靠的评估指标。在AnyCapEval上，ACM可显著提升各类基础模型的字幕生成质量。值得注意的是，ACM-8B将GPT-4o的内容得分提升了45%，风格得分提升了12%，同时在MIA-Bench、VidCapBench等常用基准上也取得了大幅性能提升。

英文摘要：

Controllable captioning is essential for precise multimodal alignment and instruction following, yet existing models often lack fine-grained control and reliable evaluation protocols. To address this gap, we present the AnyCap Project, an integrated solution spanning model, dataset, and evaluation. We introduce AnyCapModel (ACM), a lightweight plug-and-play framework that enhances the controllability of existing foundation models for omni-modal captioning without retraining the base model. ACM reuses the original captions from base models while incorporating user instructions and modality features to generate improved captions. To remedy the data scarcity in controllable multimodal captioning, we build AnyCapDataset (ACD), covering three modalities, 28 user-instruction types, and 300\,k high-quality data entries. We further propose AnyCapEval, a new benchmark that provides more reliable evaluation metrics for controllable captioning by decoupling content accuracy and stylistic fidelity. ACM markedly improves caption quality across a diverse set of base models on AnyCapEval. Notably, ACM-8B raises GPT-4oś content scores by 45\% and style scores by 12\%, and it also achieves substantial gains on widely used benchmarks such as MIA-Bench and VidCapBench.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。