M2FN: Multi-step Modality Fusion for Advertisement Image Assessment
Seoul National University（首尔国立大学）；NAVER AI LAB, NAVER CLOVA（NAVER AI实验室，NAVER CLOVA）；Soongsil University（崇实大学）；Pusan National University（釜山国立大学）；Surromind Robotics（Surromind机器人公司）
Kyung-Wha Park, Jung-Woo Ha, JungHoon Lee, Sunyoung Kwon, Kyung-Min Kim, Byoung-Tak Zhang
Assessing advertisements, specifically on the basis of user preferences and ad quality, is crucial to the marketing industry. Although recent studies have attempted to use deep neural networks for this purpose, these studies have not utilized image-related auxiliary attributes, which include embedded text frequently found in ad images. We, therefore, investigated the influence of these attributes on ad image preferences. First, we analyzed large-scale real-world ad log data and, based on our findings, proposed a novel multi-step modality fusion network (M2FN) that determines advertising images likely to appeal to user preferences. Our method utilizes auxiliary attributes through multiple steps in the network, which include conditional batch normalization-based low-level fusion and attention-based high-level fusion. We verified M2FN on the AVA dataset, which is widely used for aesthetic image assessment, and then demonstrated that M2FN can achieve state-of-the-art performance in preference prediction using a real-world ad dataset with rich auxiliary attributes.
Twins-PainViT: Towards a Modality-Agnostic Vision Transformer Framework for Multimodal Automatic Pain Assessment using Facial Videos and fNIRS
Computational Biomedicine Laboratory of the Foundation for Research and Technology (FORTH)（希腊国家研究与技术基金会计算生物医学实验室）；Department of Electrical & Computer Engineering Hellenic Mediterranean University（希腊地中海大学电气与计算机工程系）
School of Artificial Intelligence, Sun Yat-sen University（中山大学人工智能学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
K-Space-Aware Cross-Modality Score for Synthesized Neuroimage Quality Assessment
Southern University of Science and Technology（南方科技大学）；CATL（宁德时代）；Shenzhen University（深圳大学）；Tencent Youtu Lab（腾讯云图实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Massachusetts Eye and Ear（麻省眼耳医院）；Harvard Medical School（哈佛医学院）；Florida International University（佛罗里达国际大学）
Neuroimaging is essential in brain studies for the diagnosis and identification of disease, structure, and function of the brain in its healthy and disease states. Literature shows that there are advantages of multitasking with some deep learning (DL) schemes in challenging neuroimaging applications. This study examines the feasibility of using multitasking in three different applications, including tissue segmentation, cross-modality conversion, and bias-field correction. These applications reflect five different scenarios in which multitasking is explored and 280 training and testing session…
TU Delft（代尔夫特理工大学）；Instituto Tecnologico de Costa Rica（哥斯达黎加技术学院）
Laughter is considered one of the most overt signals of joy. Laughter is well-recognized as a multimodal phenomenon but is most commonly detected by sensing the sound of laughter. It is unclear how perception and annotation of laughter differ when annotated from other modalities like video, via the body movements of laughter. In this paper we take a first step in this direction by asking if and how well laughter can be annotated when only audio, only video (containing full body movement information) or audiovisual modalities are available to annotators. We ask whether annotations of laughter a…
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；School of Information and Electronic Engineering, East China Normal University（华东师范大学信息与电子工程学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）
HER2 Expression Prediction with Flexible Multi-Modal Inputs via Dynamic Bidirectional Reconstruction
School of Computer Science and Technology, University of the Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；School of Computer Science and Technology, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（巴黎- Rocquencourt 国家信息与自动化所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒实验室）
Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；ByteDance Seed（字节跳动种子）；The Chinese University of Hong Kong（香港中文大学）；City University of Hong Kong（香港城市大学）；BIAI-ZJUT；Zhejiang University of Technology（浙江工业大学）
Traits Run Deeper: Trait-Specific Asymmetric Fusion for Multimodal Personality Assessment
Hefei University of Technology（合肥工业大学）；Intelligent Interconnected Systems Laboratory of Anhui Province（安徽省智能互联系统实验室）；Jianghuai Advanced Technology Center（江淮前沿技术中心）；Anhui Provincial Industry Innovation Center of Humanoid Robots（安徽省人形机器人产业创新中心）；Anhui Provincial Key Laboratory of Humanoid Robots（安徽省人形机器人重点实验室）
提出Traits Run Deeper框架，通过多模态基础表示、特质特异性非对称融合和分布校准回归模块，解决人格评估中模态偏好差异和标签偏差问题，在AVI Challenge 2026上MSE降低约25%。
Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment
Tsinghua University（清华大学）；Beihang University（北京航空航天大学）；Children’s Hospital, Capital Institute of Pediatrics（首都儿科研究所附属儿童医院）；Zhongguancun Laboratory（中关村实验室）
Lehigh University（理海大学）；University of Pittsburgh（匹兹堡大学）；The Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院）
The assessment of Alzheimer's Disease (AD) and Mild Cognitive Impairment (MCI) associated with brain changes remains a challenging task. Recent studies have demonstrated that combination of multi-modality imaging techniques can better reflect pathological characteristics and contribute to more accurate diagnosis of AD and MCI. In this paper, we propose a novel tensor-based multi-modality feature selection and regression method for diagnosis and biomarker identification of AD and MCI from normal controls. Specifically, we leverage the tensor structure to exploit high-level correlation informati…
TAMI: Temporally Aligned, Missingness-Aware, and Interpretable Multimodal Fusion for Mental Health Assessment in Older Adults with Mild Cognitive Impairment
Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；Cairo University（开罗大学）
MedIQA: A Scalable Foundation Model for Prompt-Driven Medical Image Quality Assessment
Faculty of Applied Sciences, Macao Polytechnic University, Macao, China（澳门理工学院）；Department of Engineer Science, University of Oxford, Oxford, UK（牛津大学工程科学系）；Great Bay University, Dongguan, China（东莞大湾大学）；College of Physics and Information Engineering, Fuzhou University, Fuzhou, China（福州大学物理与信息工程学院）；Shanghai Jiao Tong University, Shanghai, China（上海交通大学）；Department of Radiology, Shenzhen People’s Hospital, Shenzhen, China（深圳人民医院放射科）
Med-K2N: Flexible K-to-N Modality Translation for Medical Image Synthesis
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute of Biomedical Engineering and Technology（苏州生物医学工程与技术研究所）；Chinese Academy of Sciences（中国科学院）；The Third Affiliated Hospital of Sun Yat-sen University（中山大学第三附属医院）
MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment
Qiushi Academy for Advanced Studies (QAAS), Zhejiang University（浙江大学启斯特先进研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；State Key Lab of Brain-Machine Intelligence（脑机智能国家重点实验室）；Collaborative Innovation Center for Artificial Intelligence by MOE and Zhejiang Provincial Government (ZJU)（教育部-浙江省人工智能协同创新中心）；Zhejiang Lab（浙江实验室）
University of Oxford（牛津大学）；Fuzhou University（福州大学）；Fudan University（复旦大学）
Multi-modality cardiac imaging plays a key role in the management of patients with cardiovascular diseases. It allows a combination of complementary anatomical, morphological and functional information, increases diagnosis accuracy, and improves the efficacy of cardiovascular interventions and clinical outcomes. Fully-automated processing and quantitative analysis of multi-modality cardiac images could have a direct impact on clinical research and evidence-based patient management. However, these require overcoming significant challenges including inter-modality misalignment and finding optima…
University of Bristol（布里斯托大学）；Southmead Hospital（南米德医院）
Evaluating neurological disorders such as Parkinson's disease (PD) is a challenging task that requires the assessment of several motor and non-motor functions. In this paper, we present an end-to-end deep learning framework to measure PD severity in two important components, hand movement and gait, of the Unified Parkinson's Disease Rating Scale (UPDRS). Our method leverages on an Inflated 3D CNN trained by a temporal segment framework to learn spatial and long temporal structure in video data. We also deploy a temporal attention mechanism to boost the performance of our model. Further, motion…
3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding
School of Geospatial Engineering and Science（测绘工程与科学学院）；Sun Yat-sen University（中山大学）；College of Electronic Science（电子科学学院）；National University of Defense Technology（国防科技大学）；Department of Civil and Environmental Engineering（土木与环境工程系）；Norwegian University of Science and Technology（挪威科技大学）；National Geomatics Center of China（中国国家测绘地理信息中心）
Label-Efficient Cross-Modality Generalization for Liver Segmentation in Multi-Phase MRI
Ho Chi Minh University of Science（胡志明科技大学）；National Central University（国立中央大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Houston（休斯顿大学）；Northwestern University（西北大学）
University College London（伦敦大学学院）；St Bartholomew’s Hospital, NHS Foundation Trust（圣巴塞洛缪医院NHS基金会信托）；St George’s University Hospitals, NHS Foundation Trust（圣乔治大学医院NHS基金会信托）
Transoesophageal echocardiography (TEE) is a valuable diagnostic and monitoring imaging modality. Proper image acquisition is essential for diagnosis, yet current assessment techniques are solely based on manual expert review. This paper presents a supervised deep learn ing framework for automatically evaluating and grading the quality of TEE images. To obtain the necessary dataset, 38 participants of varied experience performed TEE exams with a high-fidelity virtual reality (VR) platform. Two Convolutional Neural Network (CNN) architectures, AlexNet and VGG, structured to perform regression, …
Adversarial Versus Federated: An Adversarial Learning based Multi-Modality Cross-Domain Federated Medical Segmentation
Department of Electronics and Information Engineering, Beihang University（电子信息工程系，北航）；Department of Gynecology and Obstetrics, Beijing Shijitan Hospital, Capital Medical University（妇产科，北京什刹海医院，首都医科大学）；Department of Computer Science, University of Liverpool（计算机科学系，利物浦大学）；School of Computing and Mathematical Sciences, University of Leicester（计算与数学科学学院，莱斯特大学）
Towards a vision foundation model for comprehensive assessment of Cardiac MRI
Siemens Healthineers（西门子医疗）；Klinikum rechts der Isar, Technical University of Munich（慕尼黑工业大学 rechts der Isar 医院）；Siemens Healthineers AG（西门子医疗股份公司）；Imperial College London（伦敦帝国学院）
ArtiMuse: Fine-Grained Image Aesthetics Assessment with Joint Scoring and Expert-Level Understanding
USTC（中国科学技术大学）；Shanghai AI Lab（上海人工智能实验室）；China Academy of Art（中国美术学院）；Peking University（北京大学）；SJTU（上海交通大学）；Sun Yat-sen University（中山大学）；CUHK（香港中文大学）；Hong Kong PolyU（香港理工大学）
Baidu Inc.（百度公司）；Zhongshan Ophthalmic Center, Sun Yat-sen University（中山大学中山眼科中心）；Xiamen University（厦门大学）
Color fundus photography and Optical Coherence Tomography (OCT) are the two most cost-effective tools for glaucoma screening. Both two modalities of images have prominent biomarkers to indicate glaucoma suspected. Clinically, it is often recommended to take both of the screenings for a more accurate and reliable diagnosis. However, although numerous algorithms are proposed based on fundus images or OCT volumes in computer-aided diagnosis, there are still few methods leveraging both of the modalities for the glaucoma assessment. Inspired by the success of Retinal Fundus Glaucoma Challenge (REFU…
CC-DCNet: Dynamic Convolutional Neural Network with Contrastive Constraints for Identifying Lung Cancer Subtypes on Multi-modality Images
Zhejiang Lab（之江实验室）；Institute of Computer Graphics and Vision（计算机图形与视觉研究所）；Graz University of Technology（格拉茨工业大学）；School of Computer Science and Engineering（计算机科学与工程学院）；Northwestern Polytechnical University（西北工业大学）；Zhejiang University School of Medicine（浙江大学医学院）；Sir Run Run Shaw Hospital（邵逸夫医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
PAII Inc.（PAII公司）；Chang Gung Memorial Hospital（长庚纪念医院）；Vanderbilt University（范德堡大学）；PingAn Technology（平安科技）
Ultrasound (US) is a critical modality for diagnosing liver fibrosis. Unfortunately, assessment is very subjective, motivating automated approaches. We introduce a principled deep convolutional neural network (CNN) workflow that incorporates several innovations. First, to avoid overfitting on non-relevant image features, we force the network to focus on a clinical region of interest (ROI), encompassing the liver parenchyma and upper border. Second, we introduce global heteroimage fusion (GHIF), which allows the CNN to fuse features from any arbitrary number of images in a study, increasing its…
UniSpine-GS: An Efficient Physics-Aware Gaussian Framework for Cross-Modality Multi-view Spine Image Synthesis
School of Computer Science, Wuhan University（武汉大学计算机科学学院）；The Department of Ultrasound of Renmin Hospital East Branch of Wuhan University（武汉大学人民医院东院区超声科）；Institute of Artificial Intelligence, Wuhan University（武汉大学人工智能研究院）；National Engineering Research Center for Multimedia Software, Wuhan University（武汉大学多媒体软件国家工程研究中心）；Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（武汉大学多媒体网络通信工程湖北省重点实验室）
Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Despite the widespread availability of in-treatment room cone beam computed tomography (CBCT) imaging, due to the lack of reliable segmentation methods, CBCT is only used for gross set up corrections in lung radiotherapies. Accurate and reliable auto-segmentation tools could potentiate volumetric response assessment and geometry-guided adaptive radiation therapies. Therefore, we developed a new deep learning CBCT lung tumor segmentation method. Methods: The key idea of our approach called cross modality educed distillation (CMEDL) is to use magnetic resonance imaging (MRI) to guide a CBCT segm…
Expert-Agnostic Ultrasound Image Quality Assessment using Deep Variational Clustering
All India Institute of Medical Sciences (AIIMS)（全印医学科学学院）；Purdue University (PU)（普渡大学）；Indian Institute of Technology (IIT)（印度理工学院）；Berea College（伯里亚学院）
State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University（测绘遥感信息工程国家重点实验室，武汉大学）；Electronic Information School, Wuhan University（电子信息学院，武汉大学）
Tissue oxygenation and perfusion can be an indicator for organ viability during minimally invasive surgery, for example allowing real-time assessment of tissue perfusion and oxygen saturation. Multispectral imaging is an optical modality that can inspect tissue perfusion in wide field images without contact. In this paper, we present a novel, fast method for using RGB images for MSI, which while limiting the spectral resolution of the modality allows normal laparoscopic systems to be used. We exploit the discrete Haar decomposition to separate individual video frames into low pass and directio…
AI-based Automatic Segmentation of Prostate on Multi-modality Images: A Review
Soochow University（苏州大学）；Peking University First Hospital（北京大学第一医院）；Duke Kunshan University（昆山杜克大学）；Swinburne University of Technology（斯威本科技大学）；The Hong Kong Polytechnic University（香港理工大学）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Inception Institute of Artificial Intelligence（起源人工智能研究院）；Baidu Inc.（百度公司）；Southern University of Science and Technology（南方科技大学）；Cixi Institute of Biomedical Engineering, CAS（中国科学院慈溪生物医学工程研究所）
Retinal image quality assessment (RIQA) is essential for controlling the quality of retinal imaging and guaranteeing the reliability of diagnoses by ophthalmologists or automated analysis systems. Existing RIQA methods focus on the RGB color-space and are developed based on small datasets with binary quality labels (i.e., `Accept' and `Reject'). In this paper, we first re-annotate an Eye-Quality (EyeQ) dataset with 28,792 retinal images from the EyePACS dataset, based on a three-level quality grading system (i.e., `Good', `Usable' and `Reject') for evaluating RIQA methods. Our RIQA dataset is …
FunPiQ: A New Benchmark for Pixel-Level Quality Assessment in Fundus Images
Institute of Artificial Intelligence, Center for Medical Data Science, Medical University of Vienna（维也纳医科大学医学数据科学中心人工智能研究所）；Christian Doppler Lab for Artificial Intelligence in Retina, Center for Medical Data Science, Medical University of Vienna（维也纳医科大学医学数据科学中心视网膜人工智能克里斯蒂安·多普勒实验室）
Can BEV Perception Gracefully Degrade under Sensor Failures?
Tianjin Key Laboratory of Intelligent Unmanned Swarm Technology and System（天津智能无人群技术与系统重点实验室）；School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Key Laboratory of System Control and Information Processing, Ministry of Education of China（系统控制与信息处理重点实验室，中华人民共和国教育部）
University of Cambridge（剑桥大学）；Beijing Institute of Technology（北京理工大学）
Whole slide images (WSI) provide valuable phenotypic information for histological assessment and malignancy grading of tumors. The WSI-based grading promises to provide rapid diagnostic support and facilitate digital health. Currently, the most commonly used WSIs are derived from formalin-fixed paraffin-embedded (FFPE) and Frozen section. The majority of automatic tumor grading models are developed based on FFPE sections, which could be affected by the artifacts introduced by tissue processing. The frozen section exists problems such as low quality that might influence training within single m…
