The absence of food monitoring has contributed significantly to the increase in the population's weight. Due to the lack of time and busy routines, most people do not control and record what is consumed in their diet. Some solutions have been proposed in computer vision to recognize food images, but few are specialized in nutritional monitoring. This work presents the development of an intelligent system that classifies and segments food presented in images to help the automatic monitoring of user diet and nutritional intake. This work shows a comparative study of state-of-the-art methods for image classification and segmentation, applied to food recognition. In our methodology, we compare the FCN, ENet, SegNet, DeepLabV3+, and Mask RCNN algorithms. We build a dataset composed of the most consumed Brazilian food types, containing nine classes and a total of 1250 images. The models were evaluated using the following metrics: Intersection over Union, Sensitivity, Specificity, Balanced Precision, and Positive Predefined Value. We also propose an system integrated into a mobile application that automatically recognizes and estimates the nutrients in a meal, assisting people with better nutritional monitoring. The proposed solution showed better results than the existing ones in the market. The dataset is publicly available at the following link http://doi.org/10.5281/zenodo.4041488
Understanding the nutritional content of food from visual data is a challenging computer vision problem, with the potential to have a positive and widespread impact on public health. Studies in this area are limited to existing datasets in the field that lack sufficient diversity or labels required for training models with nutritional understanding capability. We introduce Nutrition5k, a novel dataset of 5k diverse, real world food dishes with corresponding video streams, depth images, component weights, and high accuracy nutritional content annotation. We demonstrate the potential of this dat…
OmniFood8K: Single-Image Nutrition Estimation via Hierarchical Frequency-Aligned Fusion
School of Software, Yunnan University, China（云南大学软件学院，中国）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences, China（中国科学院人工智能安全国家重点实验室，计算技术研究所，中国）；University of Chinese Academy of Sciences, China（中国科学院大学，中国）
A rapidly growing amount of content posted online, such as food recipes, opens doors to new exciting applications at the intersection of vision and language. In this work, we aim to estimate the calorie amount of a meal directly from an image by learning from recipes people have published on the Internet, thus skipping time-consuming manual data annotation. Since there are few large-scale publicly available datasets captured in unconstrained environments, we propose the pic2kcal benchmark comprising 308,000 images from over 70,000 recipes including photographs, ingredients and instructions. To…
RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Models
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）
University of Groningen（格罗宁根大学）；University of Barcelona（巴塞罗那大学）；University of Rovira i Virgili（罗维拉-威尔吉利大学）
Recent studies have shown that the environment where people eat can affect their nutritional behaviour. In this work, we provide automatic tools for a personalised analysis of a person's health habits by the examination of daily recorded egocentric photo-streams. Specifically, we propose a new automatic approach for the classification of food-related environments, that is able to classify up to 15 such scenes. In this way, people can monitor the context around their food intake in order to get an objective insight into their daily eating routine. We propose a model that classifies food-related…
ARTORG Center for Biomedical Engineering Research, University of Bern（伯尔尼大学ARTORG生物医学工程研究中心）；University of Maryland School of Medicine（马里兰大学医学院）；Inselspital, University Hospital, University of Bern（伯尔尼大学附属医院 Inselspital）
University of Waterloo（滑铁卢大学）；Schlegel-UW Research Institute for Aging（施莱格尔-滑铁卢大学老龄化研究所）
Dysphagia affects 590 million people worldwide and increases risk for malnutrition. Pureed food may reduce choking, however preparation differences impact nutrient density making quality assurance necessary. This paper is the first study to investigate the feasibility of computational pureed food nutritional density analysis using an imaging system. Motivated by a theoretical optical dilution model, a novel deep neural network (DNN) was evaluated using 390 samples from thirteen types of commercially prepared purees at five dilutions. The DNN predicted relative concentration of the puree sample…
Bernoulli Institute, University of Groningen（格罗宁根大学伯努利研究所）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
Eating habits are learned throughout the early stages of our lives. However, it is not easy to be aware of how our food-related routine affects our healthy living. In this work, we address the unsupervised discovery of nutritional habits from egocentric photo-streams. We build a food-related behavioural pattern discovery model, which discloses nutritional routines from the activities performed throughout the days. To do so, we rely on Dynamic-Time-Warping for the evaluation of similarity among the collected days. Within this framework, we present a simple, but robust and fast novel classificat…
University of Lincoln（林肯大学）；National Centre for Food Manufacturing（国家食品制造中心）；University of Aberdeen（阿伯丁大学）
Retail food packaging contains information which informs choice and can be vital to consumer health, including product name, ingredients list, nutritional information, allergens, preparation guidelines, pack weight, storage and shelf life information (use-by / best before dates). The presence and accuracy of such information is critical to ensure a detailed understanding of the product and to reduce the potential for health risks. Consequently, erroneous or illegible labeling has the potential to be highly detrimental to consumers and many other stakeholders in the supply chain. In this paper,…
Universidad Católica del Norte（北天主教大学）；Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
The increase in awareness of people towards their nutritional habits has drawn considerable attention to the field of automatic food analysis. Focusing on self-service restaurants environment, automatic food analysis is not only useful for extracting nutritional information from foods selected by customers, it is also of high interest to speed up the service solving the bottleneck produced at the cashiers in times of high demand. In this paper, we address the problem of automatic food tray analysis in canteens and restaurants environment, which consists in predicting multiple foods placed on a…
National University of Singapore（新加坡国立大学）；Ng Teng Fong General Hospital（黄廷芳综合医院）；Universität Stuttgart（斯图加特大学）；TikTok Singapore（TikTok新加坡）；National University Heart Centre, Singapore（新加坡国立大学心脏中心）
Singapore has been striving to improve the provision of healthcare services to her people. In this course, the government has taken note of the deficiency in regulating and supervising people's nutrient intake, which is identified as a contributing factor to the development of chronic diseases. Consequently, this issue has garnered significant attention. In this paper, we share our experience in addressing this issue and attaining medical-grade nutrient intake information to benefit Singaporeans in different aspects. To this end, we develop the FoodSG platform to incubate diverse healthcare-or…
Universitat de Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
One of the most common critical factors directly related to the cause of a chronic disease is unhealthy diet consumption. In this sense, building an automatic system for food analysis could allow a better understanding of the nutritional information with respect to the food eaten and thus it could help in taking corrective actions in order to consume a better diet. The Computer Vision community has focused its efforts on several areas involved in the visual food analysis such as: food detection, food recognition, food localization, portion estimation, among others. For food detection, the best…
The University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；The University of Massachusetts Medical School（马萨诸塞大学医学院）
Worldwide, in 2014, more than 1.9 billion adults, 18 years and older, were overweight. Of these, over 600 million were obese. Accurately documenting dietary caloric intake is crucial to manage weight loss, but also presents challenges because most of the current methods for dietary assessment must rely on memory to recall foods eaten. The ultimate goal of our research is to develop computer-aided technical solutions to enhance and improve the accuracy of current measurements of dietary intake. Our proposed system in this paper aims to improve the accuracy of dietary assessment by analyzing the…
DPF-Nutrition: Food Nutrition Estimation via Depth Prediction and Fusion
School of Electronic Information and Communication, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Institute of Agricultural Products Processing and Nuclear Agricultural Technology, Hubei Academy of Agricultural Science（湖北省农业科学院农产品加工与核农技术研究所）
University of Waterloo（滑铁卢大学）；Waterloo AI Institute（滑铁卢人工智能研究所）；Schlegel-UW Research Institute for Aging（施莱格尔-滑铁卢衰老研究学院）；KITE-Toronto Rehabilitation Institute（多伦多康复研究所KITE）；University Health Network（大学健康网络）
Malnutrition is a multidomain problem affecting 54% of older adults in long-term care (LTC). Monitoring nutritional intake in LTC is laborious and subjective, limiting clinical inference capabilities. Recent advances in automatic image-based food estimation have not yet been evaluated in LTC settings. Here, we describe a fully automatic imaging system for quantifying food intake. We propose a novel deep convolutional encoder-decoder food network with depth-refinement (EDFN-D) using an RGB-D camera for quantifying a plate's remaining food volume relative to reference portions in whole and modif…
Food fraud has been an area of great concern due to its risk to public health, reduction of food quality or nutritional value and for its economic consequences. For this reason, it's been object of regulation in many countries (e.g. [1], [2]). One type of food that has been frequently object of fraud through the addition of water or an aqueous solution is bovine meat. The traditional methods used to detect this kind of fraud are expensive, time-consuming and depend on physicochemical analysis that require complex laboratory techniques, specific for each added substance. In this paper, based on…
MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results
Purdue University（普渡大学）；University of Waterloo（滑铁卢大学）；National Research Council Canada（加拿大国家研究委员会）；Universitat de Barcelona（巴塞罗那大学）；Beijing Institute of Technology（北京理工大学）；Baidu Inc.（百度公司）；XPeng Motors（小鹏汽车）；Beijing University of Posts and Telecommunications（北京邮电大学）
Deep neural network based learning approaches is widely utilized for image classification or object detection based problems with remarkable outcomes. Realtime Object state estimation of objects can be used to track and estimate the features that the object of the current frame possesses without causing any significant delay and misclassification. A system that can detect the features of such objects in the present state from camera images can be used to enhance the application of Augmented Reality for improving user experience and delivering information in a much perceptual way. The focus beh…
MFP3D: Monocular Food Portion Estimation Leveraging 3D Point Clouds
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, USA（电气与计算机工程学院，普渡大学，西拉法叶，美国）；College of Artificial Intelligence, Anhui University, Hefei, China（人工智能学院，安徽大学，合肥，中国）
Dish detection in food platters: A framework for automated diet logging and nutrition management
Infosys Center for Artificial Intelligence（印孚瑟斯人工智能中心）；Indraprastha Institute of Information Technology Delhi (IIIT-Delhi)（德里英德拉普拉斯塔信息技术学院（IIIT-Delhi））
An important aspect of health monitoring is effective logging of food consumption. This can help management of diet-related diseases like obesity, diabetes, and even cardiovascular diseases. Moreover, food logging can help fitness enthusiasts, and people who wanting to achieve a target weight. However, food-logging is cumbersome, and requires not only taking additional effort to note down the food item consumed regularly, but also sufficient knowledge of the food item consumed (which is difficult due to the availability of a wide variety of cuisines). With increasing reliance on smart devices,…
Mam-App: A Novel Parameter-Efficient Mamba Model for Apple Leaf Disease Classification
Computer Science and Engineering, Begum Rokeya University（计算机科学与工程，贝 Gum 罗基亚大学）；Computer Science, Kumamoto University（计算机科学，熊本大学）；Information and Communication Technologies, Asian Institute of Technology（信息与通信技术，亚洲理工学院）；Computer Science, Texas State University（计算机科学，德克萨斯州立大学）
Convolutional neural networks (CNNs) apply well with food image recognition due to the ability to learn discriminative visual features. Nevertheless, recognizing distorted images is challenging for existing CNNs. Hence, the study modelled a generalized specialist approach to train a quality resilient ensemble. The approach aids the models in the ensemble framework retain general skills of recognizing clean images and shallow skills of classifying noisy images with one deep expertise area on a particular distortion. Subsequently, a novel data augmentation random quality mixup (RQMixUp) is combi…
COMSATS University Islamabad（伊斯兰堡COMSATS大学）；University of Tasmania（塔斯马尼亚大学）；National University of Ireland, Galway（爱尔兰国立高威大学）
Food resources face severe damages under extraordinary situations of catastrophes such as earthquakes, cyclones, and tsunamis. Under such scenarios, speedy assessment of food resources from agricultural land is critical as it supports aid activity in the disaster hit areas. In this article, a deep learning approach is presented for the detection and segmentation of coconut tress in aerial imagery provided through the AI competition organized by the World Bank in collaboration with OpenAerialMap and WeRobotics. Maked Region-based Convolutional Neural Network approach was used identification and…
