扩散精化的VQA标注用于半监督视线跟随
Diffusion-Refined VQA Annotations for Semi-Supervised Gaze Following
AI总结：
提出首个半监督视线跟随方法，利用VQA模型的Grad-CAM热图和扩散模型精化伪标注，在GazeFollow和VideoAttentionTarget数据集上超越基线，并减少50%标注需求。
摘要：
训练视线跟随模型需要大量由人工标注者标注视线目标坐标的图像，这一过程既费力又 inherently 模糊。我们提出了首个用于视线跟随的半监督方法，通过引入两个新颖的先验。我们利用一个大型预训练的视觉问答（VQA）模型获得第一个先验，通过向VQA模型“提示”一个视线跟随问题来计算Grad-CAM热图。这些热图可能带有噪声，不适合直接用于训练。对噪声标注进行精化的需求促使我们引入第二个先验。我们利用一个在有限人工标注上训练的扩散模型，并修改反向采样过程以精化Grad-CAM热图。通过调节扩散过程，我们在人工标注先验和VQA热图先验之间实现权衡，保留有用的VQA先验信息，同时展现出与训练数据分布相似的属性。我们的方法在GazeFollow图像数据集上优于简单的伪标注生成基线。更重要的是，我们的伪标注策略应用于广泛使用的监督视线跟随模型（VAT）时，将标注需求减少了50%。我们的方法在VideoAttentionTarget数据集上也表现最佳。
英文摘要：
Training gaze following models requires a large number of images with gaze target coordinates annotated by human annotators, which is a laborious and inherently ambiguous process. We propose the first semi-supervised method for gaze following by introducing two novel priors to the task. We obtain the first prior using a large pretrained Visual Question Answering (VQA) model, where we compute Grad-CAM heatmaps by `prompting' the VQA model with a gaze following question. These heatmaps can be noisy and not suited for use in training. The need to refine these noisy annotations leads us to incorporate a second prior. We utilize a diffusion model trained on limited human annotations and modify the reverse sampling process to refine the Grad-CAM heatmaps. By tuning the diffusion process we achieve a trade-off between the human annotation prior and the VQA heatmap prior, which retains the useful VQA prior information while exhibiting similar properties to the training data distribution. Our method outperforms simple pseudo-annotation generation baselines on the GazeFollow image dataset. More importantly, our pseudo-annotation strategy, applied to a widely used supervised gaze following model (VAT), reduces the annotation need by 50%. Our method also performs the best on the VideoAttentionTarget dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
