RiVaT-Fuse：模态不确定性下用于多模态预测的可靠性校准变分张量融合

中文摘要 AI 辅助

图像-元数据预测需要融合异质证据，这些证据的可靠性可能随样本和潜在因素而变化。现有的表示级融合方法通常选择聚合架构，如拼接、门控、条件调制或注意力，而没有明确定义在模态不确定性下融合表示应具有的含义。我们提出RiVaT-Fuse，一种可靠性校准的变分张量融合框架，将融合定义为样本级潜在状态估计。RiVaT-Fuse不是通过直接聚合产生融合向量，而是通过变分目标估计共识潜在状态，该目标平衡图像证据、元数据证据、结构化跨模态交互和稳定性。所得框架用矩阵值信任几何取代标量模态置信度，将交互分解为加性、乘性和关系组件，并将潜在状态与条件鲁棒性和结构化多任务预测耦合。我们提供了潜在求解的适定性和稳定性解释，并用高效的低秩加对角信任算子实例化该框架。在图像级图像-元数据预测基准上，RiVaT-Fuse在直接表示级基线中实现了最强的整体预测排名，同时提高了扰动下的概率和标签稳定性。

英文摘要

Multimodal prediction from images and structured metadata requires integrating complementary evidence whose reliability can vary across samples and latent directions. A single confidence weight per modality cannot capture this directional variation. We propose RiVaT-Fuse, a reliability-calibrated variational tensor fusion framework that formulates fusion as sample-wise latent-state estimation. For each image-metadata pair, the fused representation minimizes a quadratic objective combining agreement with modality embeddings, structured cross-modal interactions, and regularization. Positive-definite, low-rank-plus-diagonal trust matrices are conditioned on learned state descriptors and metadata completeness, allowing modality contributions to vary across latent directions. Additive, multiplicative, and relational interactions model cross-modal dependencies within the latent estimation objective. The resulting system admits a unique solution computed through a differentiable linear solve. A first-order analysis with fixed trust operators relates latent sensitivity to system conditioning and perturbations in modality embeddings and interactions. The framework further incorporates a state-binned entropic surrogate for conditional distributionally robust learning and task-coupled quadratic prediction heads. We instantiate RiVaT-Fuse on mBRSET, pairing retinal images with clinical and demographic metadata for diabetic retinopathy grading, diabetic macular edema detection, and referable-status prediction. Comparisons with unimodal and representation-level fusion baselines assess the predictive utility of the complete framework across these related clinical tasks.

发表机构

• Oklahoma State University（俄克拉荷马州立大学）
• George Mason University（乔治梅森大学）

机构由 AI 辅助整理，请以论文原文为准。