arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.01387cs.CV
收藏

DiffAugment: 基于扩散模型的长尾视觉关系识别

DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition

Monash University(墨尔本大学);Indian Institute of Technology Ropar(印度理工学院罗帕尔分校)

Parul Gupta, Tuan Nguyen, Abhinav Dhall, Munawar Hayat, Trung Le, Thanh-Toan Do

AI总结:

针对视觉关系识别中的长尾分布问题,提出DiffAugment方法,利用WordNet和扩散模型增强尾部类别的文本与视觉表示,并引入难度感知组件和基于主客体的种子策略,有效提升了尾部类别的平均准确率。

摘要:

视觉关系识别(VRR)任务旨在识别图像中两个交互对象之间的关系,由于<主体, 关系, �体>三元组的广泛且高度不平衡分布,该任务极具挑战性。为克服现有VRR方法中的性能偏差,我们引入了DiffAugment——一种通过利用WordNet增强尾部类别在文本空间中的表示,并利用扩散模型的生成能力为少数类别扩展视觉空间的方法。我们在扩散模型中提出了一种新颖的难度感知组件,该组件基于每个<主体, 关系, 客体>三元组的难度。我们展示了难度感知扩散在为尾部类别生成视觉嵌入方面的有效性。我们还提出了一种新颖的基于主体和客体的扩散采样种子策略,该策略提高了生成视觉嵌入的判别能力。在GQA-LT数据集上的大量实验表明,使用扩散增强数据在每类平均准确率上取得了良好的提升。

英文摘要:

The task of Visual Relationship Recognition (VRR) aims to identify relationships between two interacting objects in an image and is particularly challenging due to the widely-spread and highly imbalanced distribution of <subject, relation, object> triplets. To overcome the resultant performance bias in existing VRR approaches, we introduce DiffAugment -- a method which first augments the tail classes in the linguistic space by making use of WordNet and then utilizes the generative prowess of Diffusion Models to expand the visual space for minority classes. We propose a novel hardness-aware component in diffusion which is based upon the hardness of each <S,R,O> triplet and demonstrate the effectiveness of hardness-aware diffusion in generating visual embeddings for the tail classes. We also propose a novel subject and object based seeding strategy for diffusion sampling which improves the discriminative capability of the generated visual embeddings. Extensive experimentation on the GQA-LT dataset shows favorable gains in the subject/object and relation average per-class accuracy using Diffusion augmented samples.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑