arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.02898cs.CV
收藏

一个补丁即可描述它们全部:一种统一的零样本描述框架

One Patch to Caption Them All: A Unified Zero-Shot Captioning Framework

ISTI-CNR, Italy(意大利国家研究委员会信息科学与技术研究所);University of Pisa, Italy(比萨大学)

Lorenzo Bianchi, Giacomo Pacini, Fabio Carrara, Nicola Messina, Giuseppe Amato, Fabrizio Falchi

AI总结:

本文提出一种统一的零样本描述框架,从图像中心转向补丁中心范式,使能描述任意区域而不需区域级监督。通过将单个补丁视为原子描述单元,实现对任意区域的描述,实验表明有意义的密集视觉特征对多区域描述任务至关重要。

摘要:

零样本描述器是最近提出的一种模型,利用通用空间视觉语言表示来描述图像,而无需依赖配对的图像-文本数据。为了描述图像,它们通过文本解码文本对齐的图像特征,但局限于全局表示和整幅图像描述。我们提出了一种统一的零样本描述框架,从图像中心转向补丁中心范式,使能描述任意区域而不需区域级监督。我们不依赖全局图像表示,而是将单个补丁视为原子描述单元,并将其汇总以描述任意区域,从单个补丁到非连续区域和整幅图像。我们分析了使当前潜在描述器在我们新提出的框架中起作用的关键因素。实验表明,产生有意义、密集视觉特征的backbone,如DINO,是实现多个基于区域的描述任务中最新性能的关键。与其它基线和最新竞争对手相比,我们的模型在零样本密集描述和区域集描述上表现更好。我们还引入了一个新的trace描述任务,进一步展示了补丁级语义表示在灵活描述生成中的有效性。项目页面:https://paciosoft.com/Patch-ioner/.

英文摘要:

Zero-shot captioners are recently proposed models that utilize common-space vision-language representations to caption images without relying on paired image-text data. To caption an image, they proceed by textually decoding a text-aligned image feature, but they limit their scope to global representations and whole-image captions. We present a unified framework for zero-shot captioning that shifts from an image-centric to a patch-centric paradigm, enabling the captioning of arbitrary regions without the need of region-level supervision. Instead of relying on global image representations, we treat individual patches as atomic captioning units and aggregate them to describe arbitrary regions, from single patches to non-contiguous areas and entire images. We analyze the key ingredients that enable current latent captioners to work in our novel proposed framework. Experiments demonstrate that backbones producing meaningful, dense visual features, such as DINO, are key to achieving state-of-the-art performance in multiple region-based captioning tasks. Compared to other baselines and state-of-the-art competitors, our models achieve better performance on zero-shot dense captioning and region-set captioning. We also introduce a new trace captioning task that further demonstrates the effectiveness of patch-wise semantic representations for flexible caption generation. Project page at https://paciosoft.com/Patch-ioner/ .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑