With expansion of the video advertising market, research to predict the effects of video advertising is getting more attention. Although effect prediction of image advertising has been explored a lot, prediction for video advertising is still challenging with seldom research. In this research, we propose a method for predicting the click through rate (CTR) of video advertisements and analyzing the factors that determine the CTR. In this paper, we demonstrate an optimized framework for accurately predicting the effects by taking advantage of the multimodal nature of online video advertisements including video, text, and metadata features. In particular, the two types of metadata, i.e., categorical and continuous, are properly separated and normalized. To avoid overfitting, which is crucial in our task because the training data are not very rich, additional regularization layers are inserted. Experimental results show that our approach can achieve a correlation coefficient as high as 0.695, which is a significant improvement from the baseline (0.487).
A Multimodal Cross-View Model for Predicting Postoperative Neck Pain in Cervical Spondylosis Patients
Chengdu Institute of Computer Applications, Chinese Academy of Sciences（成都计算机应用研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Department of Neurosurgery, the First Medical Center, Chinese PLA General Hospital（中国人民解放军总医院神经外科部）；School of Medicine, Nankai University（南开大学医学院）
Preferred Networks；The University of Tokyo（东京大学）；RIKEN（理化学研究所）
This study addresses generating counterfactual explanations with multimodal information. Our goal is not only to classify a video into a specific category, but also to provide explanations on why it is not categorized to a specific class with combinations of visual-linguistic information. Requirements that the expected output should satisfy are referred to as counterfactuality in this paper: (1) Compatibility of visual-linguistic explanations, and (2) Positiveness/negativeness for the specific positive/negative class. Exploiting a spatio-temporal region (tube) and an attribute as visual and li…
Learning from Limited and Incomplete Data: A Multimodal Framework for Predicting Pathological Response in NSCLC
Department of Medicine and Surgery（医学与外科系）；Department of Diagnostics and Intervention, Radiation Physics, Biomedical Engineering（诊断与介入系，放射物理，生物医学工程）；Umeå University, Umeå, Sweden（乌梅大学，瑞典）
ViASNet: A Video Ad Saliency Network for Predicting Dynamic Saliency and Viewer Engagement
Department of Mathematics, University of Maryland, College Park, MD 20742, USA（数学系，马里兰大学，学院公园，MD 20742, 美国）；Robert H. Smith School of Business, University of Maryland, College Park, MD 20742, USA（罗伯特·H·史密斯商学院，马里兰大学，学院公园，MD 20742, 美国）
Singapore University of Technology and Design（新加坡科技设计大学）
The goal of this study is to develop and analyze multimodal models for predicting experienced affective responses of viewers watching movie clips. We develop hybrid multimodal prediction models based on both the video and audio of the clips. For the video content, we hypothesize that both image content and motion are crucial features for evoked emotion prediction. To capture such information, we extract features from RGB frames and optical flow using pre-trained neural networks. For the audio model, we compute an enhanced set of low-level descriptors including intensity, loudness, cepstrum, li…
There is a growing trend in placing video advertisements on social platforms for online marketing, which demands automatic approaches to understand the contents of advertisements effectively. Taking the 2021 TAAC competition as an opportunity, we developed a multimodal system to improve the ability of structured analysis of advertising video content. In our framework, we break down the video structuring analysis problem into two tasks, i.e., scene segmentation and multi-modal tagging. In scene segmentation, we build upon a temporal convolution module for temporal modeling to predict whether ad…
Predicting Immune Biomarkers with MultiModal Mixture-of-Expert Pathology Foundation Models Empowers Precision Oncology
Program of Computational Biology and Bioinforamtics, Yale University（耶鲁大学计算生物学与生物信息学项目）；Broad Institute of MIT and Harvard（麻省理工学院与哈佛大学博德研究所）；Department of Statistics and Data Science, Northwestern University（西北大学统计与数据科学系）；Department of Computer Science, Northeastern University（东北大学计算机科学系）；Department of Computer Science, Harvard University（哈佛大学计算机科学系）；Department of Pathology, Yale University（耶鲁大学病理学系）；Department of Anatomic Pathology and Laboratory Medicine, Hospital of the University of Pennsylvania（宾夕法尼亚大学医院解剖病理学与检验医学系）；Department of Pathology and Laboratory Medicine, University of California, San Francisco（加州大学旧金山分校病理学与检验医学系）；Department of Pathology and Laboratory Medicine, KK Women’s and Children’s Hospital（竹脚妇幼医院病理学与检验医学系）；Department of Biostatistics, Epidemiology and Informatics, Perelman School of Medicine, University of Pennsylvania（宾夕法尼亚大学佩雷尔曼医学院生物统计学、流行病学与信息学系）
Here, we develop an audiovisual deep residual network for multimodal apparent personality trait recognition. The network is trained end-to-end for predicting the Big Five personality traits of people from their videos. That is, the network does not require any feature engineering or visual analysis such as face detection, face landmark alignment or facial expression recognition. Recently, the network won the third place in the ChaLearn First Impressions Challenge with a test accuracy of 0.9109.
To study neurodegenerative diseases, longitudinal studies are carried on volunteer patients. During a time span of several months to several years, they go through regular medical visits to acquire data from different modalities, such as biological samples, cognitive tests, structural and functional imaging. These variables are heterogeneous but they all depend on the patient's health condition, meaning that there are possibly unknown relationships between all modalities. Some information may be specific to some modalities, others may be complementary, and others may be redundant. Some data ma…
Enhancing Online Learning by Integrating Biosensors and Multimodal Learning Analytics for Detecting and Predicting Student Behavior: A Review
Department of Computer Science Engineering, Universidad Autónoma de Madrid（马德里自治大学计算机科学工程系）；Digital Futures Institute, Teachers College Columbia University（哥伦比亚大学教师学院数字未来研究所）
Multimodal Doctor-in-the-Loop: A Clinically-Guided Explainable Framework for Predicting Pathological Response in Non-Small Cell Lung Cancer
Research Unit of Computer Systems and Bioinformatics, Department of Engineering（计算机系统与生物信息学研究单位，工程系）；Research Unit of Radiation Oncology, Department of Medicine and Surgery（放射肿瘤学研究单位，医学与外科系）；Research Unit of Anatomical Pathology, Department of Medicine and Surgery（病理学研究单位，医学与外科系）
Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning
Institute of Education, National Cheng Kung University（国立成功大学教育研究所）；Institute of Intelligent System, National Yang Ming Chiao Tung University（阳明交通大学智能系统研究所）；Department of Computer Science, University at Albany, State University of New York（纽约州立大学水牛城分校计算机科学系）
As an emerging type of user-generated content, micro-video drastically enriches people's entertainment experiences and social interactions. However, the popularity pattern of an individual micro-video still remains elusive among the researchers. One of the major challenges is that the potential popularity of a micro-video tends to fluctuate under the impact of various external factors, which makes it full of uncertainties. In addition, since micro-videos are mainly uploaded by individuals that lack professional techniques, multiple types of noise could exist that obscure useful information. In…
Multimodal Recurrent Ensembles for Predicting Brain Responses to Naturalistic Movies (Algonauts 2025)
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；TU Dresden（德累斯顿技术大学）；School for Embedded and Composite AI (SECAI)（嵌入式与复合人工智能学院）；Center for Scalable Data Analytics & AI (ScaDS.AI)（可扩展数据与人工智能中心）
CREATIS（CREATIS实验室）；Université Lyon1（里昂第一大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；INSA-Lyon（里昂国立应用科学学院）；Hospices Civils de Lyon（里昂民用医院）；CarMeN（CarMeN实验室）；INRA（法国国家农业研究院）；LARIS（LARIS实验室）；Universite d’Angers（昂热大学）
Clinical outcome prediction plays an important role in stroke patient management. From a machine learning point-of-view, one of the main challenges is dealing with heterogeneous data at patient admission, i.e. the image data which are multidimensional and the clinical data which are scalars. In this paper, a multimodal convolutional neural network - long short-term memory (CNN-LSTM) based ensemble model is proposed. For each MR image module, a dedicated network provides preliminary prediction of the clinical outcome using the modified Rankin scale (mRS). The final mRS score is obtained by merg…
EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs
Shenzhen University（深圳大学）；Guangdong Laboratory of Artificial Intelligence（广东人工智能与数字经济实验室）；Queen Mary University of London（女王学院伦敦大学）；Technical University of Munich（慕尼黑技术大学）；Imperial College London（伦敦帝国学院）；Tongji University（同济大学）
The content of today's social media is becoming more and more rich, increasingly mixing text, images, videos, and audio. It is an intriguing research question to model the interplay between these different modes in attracting user attention and engagement. But in order to pursue this study of multimodal content, we must also account for context: timing effects, community preferences, and social factors (e.g., which authors are already popular) also affect the amount of feedback and reaction that social-media posts receive. In this work, we separate out the influence of these non-content factor…
Harbin Institute of Technology（哈尔滨工业大学）；Hebei University（河北大学）；Alibaba Group（阿里巴巴集团）；National University of Defense Technology（国防科技大学）
Automated tagging of video advertisements has been a critical yet challenging problem, and it has drawn increasing interests in last years as its applications seem to be evident in many fields. Despite sustainable efforts have been made, the tagging task is still suffered from several challenges, such as, efficiently feature fusion approach is desirable, but under-explored in previous studies. In this paper, we present our approach for Multimodal Video Ads Tagging in the 2021 Tencent Advertising Algorithm Competition. Specifically, we propose a novel multi-modal feature fusion framework, with …
West Virginia University（西弗吉尼亚大学）；Case Western Reserve University（凯斯西储大学）；University of Notre Dame（圣母大学）
Social media such as Instagram and Twitter have become important platforms for marketing and selling illicit drugs. Detection of online illicit drug trafficking has become critical to combat the online trade of illicit drugs. However, the legal status often varies spatially and temporally; even for the same drug, federal and state legislation can have different regulations about its legality. Meanwhile, more drug trafficking events are disguised as a novel form of advertising commenting leading to information heterogeneity. Accordingly, accurate detection of illicit drug trafficking events (ID…
An attention-based deep learning network for predicting Platinum resistance in ovarian cancer
College of Medicine and Biological Information Engineering, Northeastern University（东北大学医学与生物信息工程学院）；Shengjing Hospital of China Medical University（中国医科大学附属盛京医院）
Planning for Cooler Cities: A Multimodal AI Framework for Predicting and Mitigating Urban Heat Stress through Urban Landscape Transformation
Department of City and Regional Planning, University of Pennsylvania（城市与区域规划系，宾夕法尼亚大学）；Guangdong Key Laboratory for Urban Informatics, Guangdong-Hong Kong-Macao Joint Laboratory for Smart Cities（广东城市信息关键实验室、粤港澳大湾区智慧城市联合实验室）；Shenzhen Key Laboratory of Spatial Information Smart Sensing（深圳空间信息智能感知关键实验室）；Department of Urban Informatics, School of Architecture and Urban Planning, Shenzhen University（城市信息系，深圳大学）；College of Big Data and Internet, Shenzhen Technology University（大数据与互联网学院，深圳科技大学）
Predicting Brain Responses To Natural Movies With Multimodal LLMs
Medical AI Research Center (MedARC)（医学人工智能研究中心（MedARC））；Psychological and Brain Sciences, Dartmouth College（心理学与脑科学系，达特茅斯学院）；Core for Advanced Magnetic Resonance Imaging (CAMRI), Baylor College of Medicine（先进磁共振成像核心（CAMRI），贝勒医学院）；Sophont；Princeton Neuroscience Institute（普林斯顿神经科学研究所）
There is increasing interest in the use of multimodal data in various web applications, such as digital advertising and e-commerce. Typical methods for extracting important information from multimodal data rely on a mid-fusion architecture that combines the feature representations from multiple encoders. However, as the number of modalities increases, several potential problems with the mid-fusion model structure arise, such as an increase in the dimensionality of the concatenated multimodal features and missing modalities. To address these problems, we propose a new concept that considers mul…
Stacked Regression using Off-the-shelf, Stimulus-tuned and Fine-tuned Neural Networks for Predicting fMRI Brain Responses to Movies (Algonauts 2025 Report)
Université Paris Cité（巴黎Cité大学）；University of Oxford（牛津大学）；University of Turin（都灵大学）；Universität Leipzig（莱比锡大学）；Max Planck School of Cognition（马克斯·普朗克认知科学学院）
Comparing Baseline and Day-1 Diffusion MRI Using Multimodal Deep Embeddings for Stroke Outcome Prediction
1 Department of Computer Science, University of Geneva, Switzerland 2 Department of Electrical \& Computer Engineering, University of Florida, FL, USA 3 Service of Medical Informatics, University Hospital of Geneva, Switzerland 4 Department of Imaging；Medical Informatics, University of Geneva, Switzerland
