We aim to estimate food portion size, a property that is strongly related to the presence of food object in 3D space, from single monocular images under real life setting. Specifically, we are interested in end-to-end estimation of food portion size, which has great potential in the field of personal health management. Unlike image segmentation or object recognition where annotation can be obtained through large scale crowd sourcing, it is much more challenging to collect datasets for portion size estimation since human cannot accurately estimate the size of an object in an arbitrary 2D image without expert knowledge. To address such challenge, we introduce a real life food image dataset collected from a nutrition study where the groundtruth food energy (calorie) is provided by registered dietitians, and will be made available to the research community. We propose a deep regression process for portion size estimation by combining features estimated from both RGB and learned energy distribution domains. Our estimates of food energy achieved state-of-the-art with a MAPE of 11.47%, significantly outperforms non-expert human estimates by 27.56%.
University of Waterloo（滑铁卢大学）；Waterloo AI Institute（滑铁卢人工智能研究所）；Schlegel-UW Research Institute for Aging（施莱格尔-滑铁卢衰老研究学院）；KITE-Toronto Rehabilitation Institute（多伦多康复研究所KITE）；University Health Network（大学健康网络）
Malnutrition is a multidomain problem affecting 54% of older adults in long-term care (LTC). Monitoring nutritional intake in LTC is laborious and subjective, limiting clinical inference capabilities. Recent advances in automatic image-based food estimation have not yet been evaluated in LTC settings. Here, we describe a fully automatic imaging system for quantifying food intake. We propose a novel deep convolutional encoder-decoder food network with depth-refinement (EDFN-D) using an RGB-D camera for quantifying a plate's remaining food volume relative to reference portions in whole and modif…
Purdue University（普渡大学）；Curtin University（科廷大学）；University of Hawaii Cancer Center（夏威夷大学癌症中心）
Deep learning based methods have achieved impressive results in many applications for image-based diet assessment such as food classification and food portion size estimation. However, existing methods only focus on one task at a time, making it difficult to apply in real life when multiple tasks need to be processed together. In this work, we propose an end-to-end multi-task framework that can achieve both food classification and food portion size estimation. We introduce a food image dataset collected from a nutrition study where the groundtruth food portion is provided by registered dietiti…
Due to the growing concern of chronic diseases and other health problems related to diet, there is a need to develop accurate methods to estimate an individual's food and energy intake. Measuring accurate dietary intake is an open research problem. In particular, accurate food portion estimation is challenging since the process of food preparation and consumption impose large variations on food shapes and appearances. In this paper, we present a food portion estimation method to estimate food energy (kilocalories) from food images using Generative Adversarial Networks (GAN). We introduce the c…
MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders
S-Lab, Nanyang Technological University（新加坡南洋理工大学S-Lab）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；University of Chinese Academy of Sciences（中国科学院大学）
Purdue University（普渡大学）；Curtin University（科廷大学）；University of Hawaii Cancer Center（夏威夷大学癌症中心）
Modern deep learning techniques have enabled advances in image-based dietary assessment such as food recognition and food portion size estimation. Valuable information on the types of foods and the amount consumed are crucial for prevention of many chronic diseases. However, existing methods for automated image-based food analysis are neither end-to-end nor are capable of processing multiple tasks (e.g., recognition and portion estimation) together, making it difficult to apply to real life applications. In this paper, we propose an image-based food analysis framework that integrates food loca…
Imperial College London（伦敦帝国理工学院）；Shanghai Jiao Tong University（上海交通大学）
Malnutrition is a major public health concern in low-and-middle-income countries (LMICs). Understanding food and nutrient intake across communities, households and individuals is critical to the development of health policies and interventions. To ease the procedure in conducting large-scale dietary assessments, we propose to implement an intelligent passive food intake assessment system via egocentric cameras particular for households in Ghana and Uganda. Algorithms are first designed to remove redundant images for minimising the storage memory. At run time, deep learning-based semantic segme…
DPF-Nutrition: Food Nutrition Estimation via Depth Prediction and Fusion
School of Electronic Information and Communication, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Institute of Agricultural Products Processing and Nuclear Agricultural Technology, Hubei Academy of Agricultural Science（湖北省农业科学院农产品加工与核农技术研究所）
MFP3D: Monocular Food Portion Estimation Leveraging 3D Point Clouds
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, USA（电气与计算机工程学院，普渡大学，西拉法叶，美国）；College of Artificial Intelligence, Anhui University, Hefei, China（人工智能学院，安徽大学，合肥，中国）
The increasing prevalence of diet-related chronic diseases coupled with the ineffectiveness of traditional diet management methods have resulted in a need for novel tools to accurately and automatically assess meals. Recently, computer vision based systems that use meal images to assess their content have been proposed. Food portion estimation is the most difficult task for individuals assessing their meals and it is also the least studied area. The present paper proposes a three-stage system to calculate portion sizes using two images of a dish acquired by mobile devices. The first stage cons…
ARTORG Center for Biomedical Engineering Research, University of Bern（伯尔尼大学ARTORG生物医学工程研究中心）
Food volume estimation is an essential step in the pipeline of dietary assessment and demands the precise depth estimation of the food surface and table plane. Existing methods based on computer vision require either multi-image input or additional depth maps, reducing convenience of implementation and practical significance. Despite the recent advances in unsupervised depth estimation from a single image, the achieved performance in the case of large texture-less areas needs to be improved. In this paper, we propose a network architecture that jointly performs geometric understanding (i.e., d…
VolTex: Food Volume Estimation using Text-Guided Segmentation and Neural Surface Reconstruction
Universitat de Barcelona（巴塞罗那大学）；Universitat Pompeu Fabra（庞培法布拉大学）；Grup de Tecnologies Interactives (GTI)（交互技术小组）；IMUB & Institut de Neurociències, Barcelona（IMUB与巴塞罗那神经科学研究所）
The University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）；The University of Massachusetts Medical School（马萨诸塞大学医学院）
Worldwide, in 2014, more than 1.9 billion adults, 18 years and older, were overweight. Of these, over 600 million were obese. Accurately documenting dietary caloric intake is crucial to manage weight loss, but also presents challenges because most of the current methods for dietary assessment must rely on memory to recall foods eaten. The ultimate goal of our research is to develop computer-aided technical solutions to enhance and improve the accuracy of current measurements of dietary intake. Our proposed system in this paper aims to improve the accuracy of dietary assessment by analyzing the…
MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results
Purdue University（普渡大学）；University of Waterloo（滑铁卢大学）；National Research Council Canada（加拿大国家研究委员会）；Universitat de Barcelona（巴塞罗那大学）；Beijing Institute of Technology（北京理工大学）；Baidu Inc.（百度公司）；XPeng Motors（小鹏汽车）；Beijing University of Posts and Telecommunications（北京邮电大学）
Monocular depth estimation is the base task in computer vision. It has a tremendous development in the decade with the development of deep learning. But the boundary blur of the depth map is still a serious problem. Research finds the boundary blur problem is mainly caused by two factors, first, the low-level features containing boundary and structure information may loss in deeper networks during the convolution process., second, the model ignores the errors introduced by the boundary area due to the few portions of the boundary in the whole areas during the backpropagation. In order to mitig…
University of Maryland（马里兰大学）；Worcester Polytechnic Institute（伍斯特理工学院）；University of Maryland Institute for Advanced Computer Studies（马里兰大学高级计算机研究所）
Distance estimation from vision is fundamental for a myriad of robotic applications such as navigation, manipulation, and planning. Inspired by the mammal's visual system, which gazes at specific objects, we develop two novel constraints relating time-to-contact, acceleration, and distance that we call the $τ$-constraint and $Φ$-constraint. They allow an active (moving) camera to estimate depth efficiently and accurately while using only a small portion of the image. The constraints are applicable to range sensing, sensor fusion, and visual servoing. We successfully validate the proposed const…
Fish diseases in aquaculture constitute a significant hazard to nutriment security. Identification of infected fishes in aquaculture remains challenging to find out at the early stage due to the dearth of necessary infrastructure. The identification of infected fish timely is an obligatory step to thwart from spreading disease. In this work, we want to find out the salmon fish disease in aquaculture, as salmon aquaculture is the fastest-growing food production system globally, accounting for 70 percent (2.5 million tons) of the market. In the alliance of flawless image processing and machine l…
Masked Image Modeling (MIM) has recently been established as a potent pre-training paradigm. A pretext task is constructed by masking patches in an input image, and this masked content is then predicted by a neural network using visible patches as sole input. This pre-training leads to state-of-the-art performance when finetuned for high-level semantic tasks, e.g. image classification and object detection. In this paper we instead seek to learn representations that transfer well to a wide variety of 3D vision and lower-level geometric downstream tasks, such as depth prediction or optical flow …
