This paper explores the intersection of Discrete Choice Modeling (DCM) and machine learning, focusing on the integration of image data into DCM's utility functions and its impact on model interpretability. We investigate the consequences of embedding high-dimensional image data that shares isomorphic information with traditional tabular inputs within a DCM framework. Our study reveals that neural network (NN) components learn and replicate tabular variable representations from images when co-occurrences exist, thereby compromising the interpretability of DCM parameters. We propose and benchmark two methodologies to address this challenge: architectural design adjustments to segregate redundant information, and isomorphic information mitigation through source information masking and inpainting. Our experiments, conducted on a semi-synthetic dataset, demonstrate that while architectural modifications prove inconclusive, direct mitigation at the data source shows to be a more effective strategy in maintaining the integrity of DCM's interpretable parameters. The paper concludes with insights into the applicability of our findings in real-world settings and discusses the implications for future research in hybrid modeling that combines complex data modalities. Full control of tabular and image data congruence is attained by using the MIT moral machine dataset, and both inputs are merged into a choice model by deploying the Learning Multinomial Logit (L-MNL) framework.
Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization
National University of Singapore(新加坡国立大学);Harvard Medical School(哈佛医学院);Massachusetts General Hospital(麻省总医院);National Neuroscience Institute(国家神经科学研究所);Duke-NUS Medical School(杜克-新加坡国立大学医学院)
Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Data
University of Adelaide, Australia(澳大利亚阿德莱德大学);New York University Abu Dhabi, UAE(阿联酋纽约大学阿布扎克分校);University of Surrey, UK(英国萨里大学);Boai hospital of Zhongshan, China(中国中山博爱医院);University of Central Florida, USA(美国佛罗里达大学)
Robust Latent Representation Tuning for Image-text Classification
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);College of Information Science and Engineering, Ritsumeikan University(立命馆大学信息科学与工程学院)
Taming Modality Entanglement in Continual Audio-Visual Segmentation
School of Artificial Intelligence, UCAS(人工智能学院,UCAS);MAIS, Institute of Automation(自动化研究所MAIS);School of Intelligent Science and Technology, University of Science and Technolog Beijing(智能科学与技术学院,北京理工大学)
Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
The University of British Columbia(英属哥伦比亚大学);Fudan University(复旦大学);Royal College of Science, Imperial College London(伦敦帝国理工学院皇家科学学院);Dyson School of Design Engineering(戴森设计工程学院);Suzhou Institute of Biomedical Engineering and Technology (SIBET), Chinese Academy of Sciences(中国科学院苏州生物医学工程技术研究所);East China Normal University(华东师范大学)
Supervise-assisted Multi-modality Fusion Diffusion Model for PET Restoration
Beijing Institute of Technology(北京理工大学);Research and Development Center of Agricultural Bank of China(中国农业银行研发中心);Peking University(北京大学);Beijing Normal University(北京师范大学)
Robult: Leveraging Redundancy and Modality Specific Features for Robust Multimodal Learning
Siebel School of Computing and Data Science, UIUC, US(UIUC计算机与数据科学学院);Department of Electrical and Computer Engineering, UIUC, US(UIUC电气与计算机工程学院);VinUni-Illinois Smart Health Center, VinUniversity, Vietnam(Vin大学-伊利诺伊智能健康中心)
BiXFormer: A Robust Framework for Maximizing Modality Effectiveness in Multi-Modal Semantic Segmentation
Graduate School of Informatics, Nagoya University(名古屋大学信息学研究科);AI Thrust, The Hong Kong University of Science and Technology(香港科学与技术大学人工智能 thrust);INSAIT, Sofia University(索菲亚大学INSAIT)
OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation
Tsinghua University(清华大学);Rutgers University(罗格斯大学);Beijing National Research Center for Information Science and Technology (BNRist)(北京信息科学与技术国家研究中心);The University of Hong Kong(香港大学)
Analyzing Periodicity and Saliency for Adult Video Detection
Hunan University of Science and Technology(湖南科技大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);Ocean University of China(中国海洋大学)
Memory-Augmented Incomplete Multimodal Survival Prediction via Cross-Slide and Gene-Attentive Hypergraph Learning
Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算机学院);School of Software, Tsinghua University(清华大学软件学院);School of Computer Science and Engineering, UNSW Sydney(新南威尔士大学计算机科学与工程学院)
Ultrasound Report Generation with Cross-Modality Feature Alignment via Unsupervised Guidance
Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Shenzhen Institute of Advanced Technology, the Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Chair for Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich(慕尼黑工业大学计算机辅助医疗程序与增强现实教席);The Third Medical Centre of Chinese PLA General Hospital(中国人民解放军总医院第三医学中心);The Second School of Clinical Medicine, Southern Medical University(南方医科大学第二临床医学院)
Towards Stable Cross-Domain Depression Recognition under Missing Modalities
School of Future Technology, South China University of Technology(未来技术学院,华南理工大学);PengCheng Laboratory(鹏城实验室);School of Software Engineering, South China University of Technology(软件工程学院,华南理工大学);Artificial Intelligence Research Institute, Shenzhen MSU-BIT University(人工智能研究院,深圳MSU-BIT大学);Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing(粤港澳大湾区情感智能与泛在计算联合实验室);School of Computer Science and Technology, Guangdong University of Technology(计算机科学与技术学院,广东工业大学);Department of Computer Science, City University of Hong Kong(计算机科学系,香港城市大学)
PG-SAM: Prior-Guided SAM with Medical for Multi-organ Segmentation
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Xi’an Jiaotong-Liverpool University(西交利物浦大学);University of Liverpool(利物浦大学);Monash University(莫纳什大学);Shanghai Jiao Tong University(上海交通大学)
How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model
College of Computer, National University of Defense Technology(国防科技大学计算机学院);School of Computer and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院)