Food image analysis is the groundwork for image-based dietary assessment, which is the process of monitoring what kinds of food and how much energy is consumed using captured food or eating scene images. Existing deep learning-based methods learn the visual representation for downstream tasks based on human annotation of each food image. However, most food images in real life are obtained without labels, and data annotation requires plenty of time and human effort, which is not feasible for real-world applications. To make use of the vast amount of unlabeled images, many existing works focus on unsupervised or self-supervised learning of visual representations directly from unlabeled data. However, none of these existing works focus on food images, which is more challenging than general objects due to its high inter-class similarity and intra-class variance.
In this paper, we focus on the implementation and analysis of existing representative self-supervised learning methods on food images. Specifically, we first compare the performance of six selected self-supervised learning models on the Food-101 dataset. Then we analyze the pros and cons of each selected model when training on food data to identify the key factors that can help improve the performance. Finally, we propose several ideas for future work on self-supervised visual representation learning for food images.
Food recognition and recipe analysis: integrating visual content, context and external knowledge
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS(中国科学院计算技术研究所中国科学院智能信息处理重点实验室)
When Segmentation is Not Enough: Rectifying Visual-Volume Discordance Through Multisensor Depth-Refined Semantic Segmentation for Food Intake Tracking in Long-Term Care
University of Waterloo(滑铁卢大学);Waterloo AI Institute(滑铁卢人工智能研究所);Schlegel-UW Research Institute for Aging(施莱格尔-滑铁卢衰老研究学院);KITE-Toronto Rehabilitation Institute(多伦多康复研究所KITE);University Health Network(大学健康网络)
营养不良是一个多领域的问题,影响54%的老年人在长期护理(LTC)。监测LTC中的营养摄入是费力和主观的,限制了临床推断能力。在自动图像为基础的食物估计的最新进展尚未在LTC设置进行评估。在这里,我们描述了一个全自动成像系统,用于量化食物摄入量。我们提出了一种新的深度卷积编码器-解码器食物网络与深度细化(EDFN-D)使用RGB-D相机量化板的剩余食物体积相对于参考部分在整个和修改纹理的食物。我们在预先标记的UNIMIB 2016食物数据集上训练和验证了网络,并在我们的两个新的LTC启发的盘子数据集(689个盘子图像,36种独特的食物)上进行了测试。EDFN-D在IOU上进行了深度细化图切割(0.879 vs. 0.887),摄入误差远低于典型的50%(平均摄入误差百分比:-4.2%)。我们确定如何标准的分割指标是不够的,由于视觉量不一致,并包括体积差异分析,以促进系统的信任。该系统提供了更高的透明度,以增强的客观性,准确性和精确度接近人类评估员,同时避免了大量的半自动方法时间要求。这可能有助于解决目前在资源有限的LTC和医院环境中自动早期营养不良检测的短期限制效用。
Beyond Images: Adaptive Fusion of Visual and Textual Data for Food Classification
Motilal Nehru National Institute of Technology Allahabad, India(Motilal Nehru国立技术学院Allahabad分校);Indian Institute of Technology Ropar, India(印度理工学院Ropar分校)
MultiFoodhat: A potential new paradigm for intelligent food quality inspection
School of Food Science and Engineering, Central South University of Forestry and Technology(食品科学与工程学院,中南林业科技大学);School of Computer and Information, Hefei University of Technology(计算机与信息学院,合肥工业大学)
RFHNet: Relational and Frequency-Aware Hashing Network for Large-Scale Fine-Grained Food Image Retrieval
College of Computer Science and Artificial Intelligence, Ludong University(鲁东大学计算机科学与人工智能学院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
FoodSense: A Multisensory Food Dataset and Benchmark for Predicting Taste, Smell, Texture, and Sound from Images
College of Engineering and Computer Science, University of Central Florida(中央佛罗里达大学工程与计算机科学学院);College of Business Administration, University of Central Florida(中央佛罗里达大学商学院);Institute of Artificial Intelligence, University of Central Florida(中央佛罗里达大学人工智能研究所)
Simulating Personal Food Consumption Patterns using a Modified Markov Chain
Purdue University(普渡大学)
食物图像分类是基于图像的膳食评估的基础,可以预测食物类别。由于在现实生活中有许多不同的食物类别,传统的模型不能达到足够高的精度。个性化分类器旨在大大提高每个人的食物图像分类的准确性。然而,缺乏公共个人食物消费数据证明是训练这种模型的一个挑战。为了解决这个问题,我们提出了一个新的框架来模拟个人的食物消费数据模式,利用修改后的马尔可夫链模型和自监督学习的使用。我们的方法是能够创建一个准确的未来的数据模式从有限数量的初始数据,我们的模拟数据模式可以密切相关的初始数据模式。此外,我们使用Dynamic Time Warping距离和Kullback-Leibler散度作为度量标准来评估我们的方法在公共Food-101数据集上的有效性。我们的实验结果表明,与随机模拟和原始马尔可夫链方法相比,有前途的性能。
Training-Free Text-to-Image Compositional Food Generation via Prompt Grafting
Elmore Family School of Electrical and Computer Engineering, Purdue University(电气与计算机工程学院,普渡大学);Department of Systems Design Engineering, University of Waterloo(系统设计工程系,滑铁卢大学)
Understanding Image2Video Domain Shift in Food Segmentation: An Instance-level Analysis on Apples
Interdisciplinary Program in Artificial Intelligence, Seoul National University(人工智能交叉学科项目,首尔国立大学);Department of Biological Environmental Science, College of Life Science and Biotechnology, Dongguk University(生命科学与生物技术学院生物环境科学系,东国大学);Department of Food Science and Biotechnology, Dongguk University(食品科学与生物技术系,东国大学)
Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring
Imperial College London(伦敦帝国理工学院);University of Pittsburgh(匹兹堡大学);Baylor College Of Medicine(贝勒医学院);University of Ghana(加纳大学);University of Georgia(佐治亚大学);Boston University(波士顿大学)