利用基础模型增强食谱检索：数据增强视角
Enhancing Recipe Retrieval with Foundation Models: A Data Augmentation Perspective
AI总结：
本文提出利用基础模型（Llama2和SAM）进行数据增强，并引入DAR框架，通过适配器层和多层级圆损失增强食谱与图像表示，在Recipe1M数据集上大幅超越现有方法。
摘要：
在公共嵌入空间中学习食谱和食物图像表示对于跨模态食谱检索而言虽非易事，却至关重要。本文提出了一种利用基础模型进行数据增强的新视角来解决该问题。借助基础模型（即Llama2和SAM）的卓越能力，我们提出通过提取与对方相关的可对齐信息来增强食谱和食物图像。具体而言，Llama2被用于从食谱生成文本描述，旨在捕捉食物图像的视觉线索；SAM则被用于生成与食谱中关键食材对应的图像分割。为充分利用增强数据，我们引入了数据增强检索框架（DAR），以增强跨模态检索的食谱和图像表示学习。我们首先向预训练的CLIP模型注入适配器层以降低计算成本，而非对所有参数进行全面微调。此外，提出了多层级圆损失来对齐原始和增强数据对，该损失对正样本对和负样本对分配不同的惩罚。在Recipe1M数据集上，我们的DAR大幅优于所有现有方法。广泛的消融研究验证了DAR各组成部分的有效性。
英文摘要：
Learning recipe and food image representation in common embedding space is non-trivial but crucial for cross-modal recipe retrieval. In this paper, we propose a new perspective for this problem by utilizing foundation models for data augmentation. Leveraging on the remarkable capabilities of foundation models (i.e., Llama2 and SAM), we propose to augment recipe and food image by extracting alignable information related to the counterpart. Specifically, Llama2 is employed to generate a textual description from the recipe, aiming to capture the visual cues of a food image, and SAM is used to produce image segments that correspond to key ingredients in the recipe. To make full use of the augmented data, we introduce Data Augmented Retrieval framework (DAR) to enhance recipe and image representation learning for cross-modal retrieval. We first inject adapter layers to pre-trained CLIP model to reduce computation cost rather than fully fine-tuning all the parameters. In addition, multi-level circle loss is proposed to align the original and augmented data pairs, which assigns different penalties for positive and negative pairs. On the Recipe1M dataset, our DAR outperforms all existing methods by a large margin. Extensive ablation studies validate the effectiveness of each component of DAR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
