This paper determines the extent to which short textual inputs (in this case, names of dishes) can improve calorie estimation compared to an image-only baseline model and whether any improvements are statistically significant. Utilizes the TensorFlow library and the Nutrition5k dataset (curated by Google) to train both an image-only CNN and multimodal CNN that accepts both text and an image as input. The MAE of calorie estimations was reduced by 1.06 kcal from 84.76 kcal to 83.70 kcal (1.25% improvement) when using the multimodal model.
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
CMU(卡内基梅隆大学);Princeton University(普林斯顿大学);Columbia University(哥伦比亚大学);Harvard Medical School(哈佛医学院);Brigham and Women’s Hospital(布莱根妇女医院);University of Pennsylvania(宾夕法尼亚大学);University of Oregon(俄勒冈大学)
FLEX: A Largescale Multimodal, Multiview Dataset for Learning Structured Representations for Fitness Action Quality Assessment
School of Biomedical Engineering (Suzhou), USTC(中国科学技术大学苏州生物医学工程学院);Suzhou Institute of Biomedical Engineering and Technology, CAS(中国科学院苏州生物医学工程技术研究所);Institute of High-Performance Computing, A*STAR, Singapore(新加坡科技研究局高性能计算研究所);School of Psychology, Beijing Sports University(北京体育大学心理学院);School of Competitive Sports, Beijing Sports University(北京体育大学竞技体育学院);Department of Robotics, University of Michigan(密歇根大学机器人系)
A tutorial on discovering and quantifying the effect of latent causal sources of multimodal EHR data
Department of Biomedical Informatics Vanderbilt University Medical Center(生物医学信息学系范德堡大学医学中心);Department of Biomedical Informatics University of Pittsburgh(生物医学信息学系匹兹堡大学);Departments of Medicine & Biomedical Informatics Vanderbilt University Medical Center(医学与生物医学信息学系范德堡大学医学中心);Departments of Biomedical Informatics & Computer Science Vanderbilt University Medical Center & Vanderbilt University(生物医学信息学与计算机科学系范德堡大学医学中心及范德堡大学)
KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models
University of California, Berkeley(加州大学伯克利分校);Boston University(波士顿大学);Google DeepMind(谷歌DeepMind);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)
BIO-CXRNET: A Robust Multimodal Stacking Machine Learning Technique for Mortality Risk Prediction of COVID-19 Patients using Chest X-Ray Images and Clinical Data
Qatar University(卡塔尔大学);North South University(北南大学);University of South Carolina(南卡罗来纳大学);Hamad General Hospital(哈马德综合医院);AFG College with the University of Aberdeen(阿伯丁大学AFG学院)