arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.08592cs.CV
收藏

基于多模态ChatGPT的膳食评估:系统性分析

Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis

Imperial College London(帝国理工学院);The Chinese University of Hong Kong(香港中文大学)

Frank P. -W. Lo, Jianing Qiu, Zeyu Wang, Junhong Chen, Bo Xiao, Wu Yuan, Stamatia Giannarou, Gary Frost, Benny Lo

AI总结:

本研究探索多模态ChatGPT在膳食评估中的应用,发现GPT-4V无需微调即可实现高准确率食物检测,具备上下文感知的分量估算能力,可推动营养科学与膳食评估技术发展。

摘要:

传统的膳食评估方法主要基于自我报告法或在营养师监督下进行的结构化访谈。然而,这些方法往往具有主观性,可能存在不准确的问题,且耗时较长。尽管人们已经设计出基于人工智能(AI)的解决方案来实现膳食评估流程的自动化,但这些现有的AI方法在适配多种食物类型、饮食行为和文化背景的泛化能力上存在挑战,导致膳食领域的AI应用专业范围狭窄、准确性有限。近年来,为最新版ChatGPT提供支持的GPT-4V等多模态基础模型的出现,在众多研究领域的各类任务(如场景理解和图像字幕生成)中展现出变革性潜力。这些模型已证明具备出色的通用智能和准确性,能够处理多种数据模态。本研究探讨了多模态ChatGPT在膳食评估领域的应用。研究结果显示,在无需任何微调或使用食物特定数据集进行适配的情况下,GPT-4V在复杂条件下的食物检测表现优异,准确率最高可达87.5%。通过用特定语言提示(如“非洲料理”)引导模型,它可从识别大米、面包等常见主食,转向准确识别banku、ugali等地区特色菜肴。GPT-4V的另一项突出特性是其上下文感知能力。它可以利用周围物体作为尺寸参考来推断食物的分量大小,进一步提升将食物重量转换为营养成分的准确性。该模型与USDA国家营养数据库的契合度,凸显了GPT-4V在推动营养科学和膳食评估技术发展方面的潜力。

英文摘要:

Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, potentially inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, these prior AI methodologies encounter challenges in their ability to generalize across a diverse range of food types, dietary behaviors, and cultural contexts. This results in AI applications in the dietary field that possess a narrow specialization and limited accuracy. Recently, the emergence of multimodal foundation models such as GPT-4V powering the latest ChatGPT has exhibited transformative potential across a wide range of tasks (e.g., Scene understanding and image captioning) in numerous research domains. These models have demonstrated remarkable generalist intelligence and accuracy, capable of processing various data modalities. In this study, we explore the application of multimodal ChatGPT within the realm of dietary assessment. Our findings reveal that GPT-4V excels in food detection under challenging conditions with accuracy up to 87.5% without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another GPT-4V's standout feature is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further enhancing its accuracy in translating food weight into nutritional content. This alignment with the USDA National Nutrient Database underscores GPT-4V's potential to advance nutritional science and dietary assessment techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑