Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, potentially inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, these prior AI methodologies encounter challenges in their ability to generalize across a diverse range of food types, dietary behaviors, and cultural contexts. This results in AI applications in the dietary field that possess a narrow specialization and limited accuracy. Recently, the emergence of multimodal foundation models such as GPT-4V powering the latest ChatGPT has exhibited transformative potential across a wide range of tasks (e.g., Scene understanding and image captioning) in numerous research domains. These models have demonstrated remarkable generalist intelligence and accuracy, capable of processing various data modalities. In this study, we explore the application of multimodal ChatGPT within the realm of dietary assessment. Our findings reveal that GPT-4V excels in food detection under challenging conditions with accuracy up to 87.5% without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another GPT-4V's standout feature is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further enhancing its accuracy in translating food weight into nutritional content. This alignment with the USDA National Nutrient Database underscores GPT-4V's potential to advance nutritional science and dietary assessment techniques.
A review on vision-based analysis for automatic dietary assessment
Beijing Technology and Business University(北京工商大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring
Imperial College London(伦敦帝国理工学院);University of Pittsburgh(匹兹堡大学);Baylor College Of Medicine(贝勒医学院);University of Ghana(加纳大学);University of Georgia(佐治亚大学);Boston University(波士顿大学)
Clustering Egocentric Images in Passive Dietary Monitoring with Self-Supervised Learning
Imperial College London(帝国理工学院);University of Pittsburgh(匹兹堡大学);Baylor College Of Medicine(贝勒医学院);University of Ghana(加纳大学);University of Georgia(佐治亚大学);Boston University(波士顿大学);University of Alabama(阿拉巴马大学)
Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images
Department of Computer Science, New Jersey Institute of Technology(新泽西理工学院计算机科学系);Vascular and Endovascular Surgery, Robert Wood Johnson Hospital(罗伯特·伍德·约翰逊医院血管外科);Department of Data Science, New Jersey Institute of Technology(新泽西理工学院数据科学系)
本研究评估了六种视觉语言模型在慢性伤口分析任务上的表现,发现通用模型 ChatGPT 和 Claude 显著优于医学专用模型,表明广泛的多模态推理能力比领域知识更重要。