Computational food analysis (CFA) naturally requires multi-modal evidence of a particular food, e.g., images, recipe text, etc. A key to making CFA possible is multi-modal shared representation learning, which aims to create a joint representation of the multiple views (text and image) of the data. In this work we propose a method for food domain cross-modal shared representation learning that preserves the vast semantic richness present in the food data. Our proposed method employs an effective transformer-based multilingual recipe encoder coupled with a traditional image embedding architecture. Here, we propose the use of imperfect multilingual translations to effectively regularize the model while at the same time adding functionality across multiple languages and alphabets. Experimental analysis on the public Recipe1M dataset shows that the representation learned via the proposed method significantly outperforms the current state-of-the-arts (SOTA) on retrieval tasks. Furthermore, the representational power of the learned representation is demonstrated through a generative food image synthesis model conditioned on recipe embeddings. Synthesized images can effectively reproduce the visual appearance of paired samples, indicating that the learned representation captures the joint semantics of both the textual recipe and its visual content, thus narrowing the modality gap.
MRSegmentator: Multi-Modality Segmentation of 40 Classes in MRI and CT
Department of Radiology, Charité - Universitätsmedizin Berlin（柏林查理医院放射科）；Freie Universität Berlin（柏林自由大学）；Humboldt Universität zu Berlin（柏林洪堡大学）；Diagnostic Image Analysis Group, Radboud University Medical Center（拉德堡德大学医学中心诊断图像分析组）；School of Medicine and Health, Klinikum rechts der Isar, TUM University Hospital（医学与健康学院，莱比锡大学医院）；Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität (FAU) Erlangen-Nürnberg（埃朗根大学医院放射科，埃朗根-纽伦堡弗里德里希-亚历山大大学）；Laboratory for Computer Aided Medical Procedures, Technical University of Munich（慕尼黑技术大学计算机辅助医疗程序实验室）；Chair for AI in Medicine and Healthcare, Klinikum rechts der Isar, Technical University Munich（医学与健康中的AI教授职位，莱比锡大学医院，慕尼黑技术大学）
VRVis Zentrum für Virtual Reality und Visualisierung Forschungs-GmbH（VRVis虚拟现实与可视化研究有限公司）；Center for Medical Physics and Biomedical Engineering, Medical University of Vienna（维也纳医科大学医学物理与生物医学工程中心）；Center for Nuclear Medicine, Clinical Center of Serbia（塞尔维亚临床中心核医学中心）
Tumor segmentation in multimodal medical images has seen a growing trend towards deep learning based methods. Typically, studies dealing with this topic fuse multimodal image data to improve the tumor segmentation contour for a single imaging modality. However, they do not take into account that tumor characteristics are emphasized differently by each modality, which affects the tumor delineation. Thus, the tumor segmentation is modality- and task-dependent. This is especially the case for soft tissue sarcomas, where, due to necrotic tumor tissue, the segmentation differs vastly. Closing this …
SOMA-1M: A Large-Scale SAR-Optical Multi-resolution Alignment Dataset for Multi-Task Remote Sensing
School of Remote Sensing Information Engineering, Wuhan University（遥感信息工程学院，武汉大学）；Hubei LuoJia Laboratory（湖北珞珈实验室）；Technology Innovation Center for Collaborative Applications of Natural Resources Data in GBA, Ministry of Natural Resources（粤港澳大湾区自然资源数据协同应用技术创新中心，自然资源部）
