arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.08701cs.CVcs.LG
收藏

多模态不确定推断的统一框架

Unified Multimodal Uncertain Inference

Johns Hopkins University(约翰霍普金斯大学);Human Language Technology Center of Excellence(人类语言技术卓越中心)

Dengjia Zhang, Alexander Martin, William Jurayj, Kenton Murray, Benjamin Van Durme, Reno Kriz

AI总结:

本文提出UMUI框架,通过多模态数据生成校准的概率估计,引入CLUE模型实现跨模态细粒度概率推理,实验表明其在多种模态上表现优于基线模型。

摘要:

我们引入统一多模态不确定推断(UMUI),这是一个涵盖文本、音频和视频的多模态推理任务,其中模型必须根据前提在任何模态或组合下生成校准的概率假设估计。尽管不确定推断已在文本中得到探索,但扩展到其他模态的尝试仅限于单模态二元蕴含判断,没有框架支持跨模态的细粒度概率推理。为解决此问题,我们精心整理了一个由人类标注的评估集,包含音频、视觉及视听设置下的标量概率判断,并额外在现有文本和音频基准上进行评估。我们引入CLUE(校准潜在不确定性估计),结合自一致教师校准和基于分布的置信度探测,以生成校准的预测。我们证明,我们的3B参数模型在所有模态上均达到与基线模型(最高32B参数)相当或更优的性能。

英文摘要:

We introduce Unified Multimodal Uncertain Inference (UMUI), a multimodal inference task spanning text, audio, and video, where models must produce calibrated probability estimates of hypotheses conditioned on a premise in any modality or combination. While uncertain inference has been explored in text, extension to other modalities has been limited to single-modality binary entailment judgments, leaving no framework for fine-grained probabilistic reasoning in or across other modalities. To address this, we curate a human-annotated evaluation set with scalar probability judgments across audio, visual, and audiovisual settings, and additionally evaluate on existing text and audio benchmarks. We introduce CLUE (Calibrated Latent Uncertainty Estimation), which combines self-consistent teacher calibration and distribution-based confidence probing to produce calibrated predictions. We demonstrate that our 3B-parameter model achieves equivalent or stronger performance than zero-shot baselines up to 32B parameters across all modalities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑