arXiv 2604.277122026-04-30cs.CVcs.CL

具有语言信息的多模态融合用于越南语场景文本图像描述：数据集、图框架和语音注意力

Linguistically Informed Multimodal Fusion for Vietnamese Scene-Text Image Captioning: Dataset, Graph Framework, and Phonological Attention

Faculty of Information Science and Engineering（信息科学与工程学院）；University of Information Technology（信息技术大学）；Vietnam National University（越南国家大学）

Nhi Ngoc-Yen Nguyen, Anh-Duc Nguyen, Nghia Hieu Nguyen, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出HSTFG和PhonoSTFG框架，通过图拓扑分析发现跨模态边对场景文本融合有害，并构建首个大规模越南语场景文本描述数据集ViTextCaps，揭示52.8%词汇存在声调符号冲突风险。

摘要：

场景文本图像描述需要融合三种信息流——视觉特征、OCR检测文本和语言知识——以生成准确整合图像中可见文本的描述。现有融合方法将文本视为语言无关，这在越南语中失效：越南语是声调语言，声调符号改变词义，OCR错误普遍存在，词边界模糊。我们主张越南语场景文本描述需要具有语言信息的多模态融合，其中语言特定的结构知识被显式纳入融合机制。受此启发，我们提出HSTFG（异构场景文本融合图），一种具有学习空间注意力偏置的通用图融合框架，并通过拓扑分析证明跨模态图边对场景文本融合有害。基于此发现，我们设计PhonoSTFG（语音场景文本融合图），专门针对越南语语言推理进行图级融合。为支持评估，我们引入ViTextCaps，首个大规模越南语场景文本描述数据集（15,729张图像，74,970条描述），全面语言分析显示52.8%的词汇存在声调符号冲突风险。

英文摘要：

Scene-text image captioning requires fusing three information streams -- visual features, OCR-detected text, and linguistic knowledge -- to generate descriptions that faithfully integrate text visible in images. Existing fusion approaches treat text as language-agnostic, which fails for Vietnamese: a tonal language where diacritics alter word meaning, OCR errors are pervasive, and word boundaries are ambiguous. We argue that Vietnamese scene-text captioning demands \textit{linguistically informed multimodal fusion}, where language-specific structural knowledge is explicitly incorporated into the fusion mechanism. Motivated from these insights, we propose \textbf{HSTFG} (Heterogeneous Scene-Text Fusion Graph), a general-purpose graph fusion framework with learned spatial attention bias, and show through topology analysis that cross-modal graph edges are harmful for scene-text fusion. Building on this finding, we design \textbf{PhonoSTFG} (Phonological Scene-Text Fusion Graph) which specializes graph-level fusion for Vietnamese linguistic reasoning. To support evaluation, we introduce \textbf{ViTextCaps}, the first large-scale Vietnamese scene-text captioning dataset (\textbf{15{,}729} images with \textbf{74{,}970} captions), with comprehensive linguistic analysis showing that 52.8\% of the vocabulary is at risk of diacritic collision.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。