OVFoodSeg：通过图像信息引导的文本表示提升开放词汇食物图像分割
OVFoodSeg: Elevating Open-Vocabulary Food Image Segmentation via Image-Informed Textual Representation
AI总结：
本文提出开放词汇食物分割框架OVFoodSeg，通过FoodLearner和图像信息引导的文本编码器将视觉上下文融入文本嵌入，在FoodSeg103上mIoU提升4.9%。
摘要：
在食物计算领域，从图像中分割食材面临巨大挑战，原因包括同类食材存在较大的类内差异、新食材不断出现，以及大规模食物分割数据集的高昂标注成本。现有方法主要采用封闭词汇和静态文本嵌入的设置。这些方法往往难以有效处理各类食材，尤其是新型和多样化的食材。针对这些局限性，我们提出了OVFoodSeg，这是一个采用开放词汇设置并利用视觉上下文增强文本嵌入的框架。通过整合视觉语言模型（VLMs），我们的方法借助两个创新模块，即图像到文本学习器FoodLearner和图像信息引导的文本编码器（Image-Informed Text Encoder），用图像特定信息丰富文本嵌入。OVFoodSeg的训练过程分为两个阶段：FoodLearner的预训练和随后的分割学习阶段。预训练阶段赋予FoodLearner将视觉信息与专门和食物相关的文本表示对齐的能力，第二阶段则使FoodLearner和图像信息引导的文本编码器适应分割任务。通过弥补以往模型的不足，OVFoodSeg展现了显著的改进，在FoodSeg103数据集上平均交并比（mIoU）提升了4.9%，为食物图像分割树立了新的里程碑。
英文摘要：
In the realm of food computing, segmenting ingredients from images poses substantial challenges due to the large intra-class variance among the same ingredients, the emergence of new ingredients, and the high annotation costs associated with large food segmentation datasets. Existing approaches primarily utilize a closed-vocabulary and static text embeddings setting. These methods often fall short in effectively handling the ingredients, particularly new and diverse ones. In response to these limitations, we introduce OVFoodSeg, a framework that adopts an open-vocabulary setting and enhances text embeddings with visual context. By integrating vision-language models (VLMs), our approach enriches text embedding with image-specific information through two innovative modules, eg, an image-to-text learner FoodLearner and an Image-Informed Text Encoder. The training process of OVFoodSeg is divided into two stages: the pre-training of FoodLearner and the subsequent learning phase for segmentation. The pre-training phase equips FoodLearner with the capability to align visual information with corresponding textual representations that are specifically related to food, while the second phase adapts both the FoodLearner and the Image-Informed Text Encoder for the segmentation task. By addressing the deficiencies of previous models, OVFoodSeg demonstrates a significant improvement, achieving an 4.9\% increase in mean Intersection over Union (mIoU) on the FoodSeg103 dataset, setting a new milestone for food image segmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
