视觉-语言模型是否准备好进行饮食评估？探索AI驱动的食品图像识别的下一个前沿
Are Vision-Language Models Ready for Dietary Assessment? Exploring the Next Frontier in AI-Powered Food Image Recognition
AI总结：
本文评估了六种先进的视觉-语言模型在不同层次上的食品识别能力，提出了一个新的评估指标，并展示了FoodNExTDB数据库在饮食评估中的应用潜力。
摘要：
基于食品图像的自动饮食评估仍是一个挑战，需要精确的食品检测、分割和分类。视觉-语言模型（VLMs）通过整合视觉和文本推理提供了新的可能性。在本研究中，我们评估了六种最先进的VLMs（ChatGPT、Gemini、Claude、Moondream、DeepSeek和LLaVA），分析它们在不同层次上的食品识别能力。在实验框架中，我们引入了FoodNExTDB，一个独特的食品图像数据库，包含9,263张由专家标注的图像，涵盖10个类别（例如“蛋白质来源”）、62个子类别（例如“家禽”）和9种烹饪风格（例如“烤制”）。总共，FoodNExTDB包括50,000个由七位专家生成的营养标签，这些标签由手动标注所有数据库中的图像生成。此外，我们提出了一种新的评估指标，专家加权召回率（EWR），该指标考虑了不同标注者之间的差异。结果表明，封闭源模型在识别包含单一产品的图像中的食品产品时，性能优于开源模型，达到了超过90%的EWR。尽管有潜力，当前VLMs在细粒度食品识别方面面临挑战，特别是在区分烹饪风格的细微差异和视觉相似的食品项目时，这限制了它们在自动饮食评估中的可靠性。FoodNExTDB数据库在https://github.com/AI4Food/FoodNExtDB上公开可用。
英文摘要：
Automatic dietary assessment based on food images remains a challenge, requiring precise food detection, segmentation, and classification. Vision-Language Models (VLMs) offer new possibilities by integrating visual and textual reasoning. In this study, we evaluate six state-of-the-art VLMs (ChatGPT, Gemini, Claude, Moondream, DeepSeek, and LLaVA), analyzing their capabilities in food recognition at different levels. For the experimental framework, we introduce the FoodNExTDB, a unique food image database that contains 9,263 expert-labeled images across 10 categories (e.g., "protein source"), 62 subcategories (e.g., "poultry"), and 9 cooking styles (e.g., "grilled"). In total, FoodNExTDB includes 50k nutritional labels generated by seven experts who manually annotated all images in the database. Also, we propose a novel evaluation metric, Expert-Weighted Recall (EWR), that accounts for the inter-annotator variability. Results show that closed-source models outperform open-source ones, achieving over 90% EWR in recognizing food products in images containing a single product. Despite their potential, current VLMs face challenges in fine-grained food recognition, particularly in distinguishing subtle differences in cooking styles and visually similar food items, which limits their reliability for automatic dietary assessment. The FoodNExTDB database is publicly available at https://github.com/AI4Food/FoodNExtDB.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
