缓解跨模态表示偏差用于多文化图像到食谱检索
Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval
AI总结：
本文提出因果表示学习方法，预测图像中遗漏的烹饪元素并注入跨模态学习，以缓解图像到食谱检索中的表示偏差，在单语和多文化数据集上显著提升检索性能。
摘要：
现有的图像到食谱检索方法隐含假设食物图像能够完全捕捉食谱中文本记录的细节。然而，食物图像仅反映烹饪菜肴的视觉结果，而非潜在的烹饪过程。因此，学习跨模态表示以弥合图像与食谱之间的模态差距，往往会忽略那些视觉上不明显但对食谱检索至关重要的、食谱特有的细微细节。具体而言，表示偏向于捕捉主导视觉元素，导致在排序使用成分和烹饪方法有细微差异的相似食谱时出现困难。当训练数据混合来自不同菜系的图像和食谱时，表示学习中的偏差预计会更加严重。本文提出了一种新颖的因果方法，预测图像中可能被忽略的烹饪元素，同时将这些元素显式注入跨模态表示学习以缓解偏差。实验在标准单语Recipe1M数据集和新构建的多语言多文化菜系数据集上进行。结果表明，所提出的因果表示学习能够揭示细微的成分和烹饪动作，并在单语和多语言多文化数据集上均取得了令人印象深刻的检索性能。
英文摘要：
Existing approaches for image-to-recipe retrieval have the implicit assumption that a food image can fully capture the details textually documented in its recipe. However, a food image only reflects the visual outcome of a cooked dish and not the underlying cooking process. Consequently, learning cross-modal representations to bridge the modality gap between images and recipes tends to ignore subtle, recipe-specific details that are not visually apparent but are crucial for recipe retrieval. Specifically, the representations are biased to capture the dominant visual elements, resulting in difficulty in ranking similar recipes with subtle differences in use of ingredients and cooking methods. The bias in representation learning is expected to be more severe when the training data is mixed of images and recipes sourced from different cuisines. This paper proposes a novel causal approach that predicts the culinary elements potentially overlooked in images, while explicitly injecting these elements into cross-modal representation learning to mitigate biases. Experiments are conducted on the standard monolingual Recipe1M dataset and a newly curated multilingual multicultural cuisine dataset. The results indicate that the proposed causal representation learning is capable of uncovering subtle ingredients and cooking actions and achieves impressive retrieval performance on both monolingual and multilingual multicultural datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
