VaaWIT: 面向多语言网页图像翻译的大语言模型视觉感知适配
VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image Translation
AI总结：
针对网页图像翻译中视觉表示差距问题，提出VaaWIT框架，通过双流注意力模块和视觉感知适配器，实现大语言模型对细粒度视觉特征的动态融合，在多个基准上超越开源模型并接近闭源模型性能。
摘要：
翻译网页图像中的文本对于改善内容可访问性和跨语言信息检索至关重要，尤其是在社交媒体和电子商务领域。尽管大型视觉语言模型（LVLMs）已经推进了多模态理解，但由于视觉表示差距，将它们应用于网页图像翻译仍然具有挑战性：标准编码器通常优先考虑高级语义，而忽略了识别多样字符形态所需的细粒度视觉细节。为了解决这一挑战，我们提出了VaaWIT，一个端到端框架，用于适配大语言模型进行多语言网页图像翻译。该框架引入了两项关键技术贡献：（1）双流注意力模块（DSAM），促进多语言语义特征与详细视觉表示之间的双向交互，从而合成对文本变化鲁棒的统一特征；（2）视觉感知适配器（VAA），一种参数高效的微调策略，将这些融合的视觉线索动态注入冻结的LLM主干。这种设计使模型能够有效地将视觉上下文与语言推理对齐，同时最小化计算成本。在三个公共基准上的八个任务上的大量实验表明，VaaWIT显著优于最先进（SOTA）的开源基线，并达到了与专有模型相竞争的性能。这些结果验证了将细粒度视觉感知集成到LLM中用于复杂网页内容分析的有效性。
英文摘要：
Translating text embedded in Web images is crucial for improving content accessibility and cross-lingual information retrieval, particularly within social media and e-commerce domains. Although Large Vision-Language Models (LVLMs) have advanced multimodal understanding, applying them to Web image translation remains challenging due to the visual representation gap: standard encoders often prioritize high-level semantics over the fine-grained visual details required for recognizing diverse character morphologies. To address this challenge, we propose VaaWIT, an end-to-end framework that adapts Large Language Models for multilingual Web image translation. The framework introduces two key technical contributions: (1) a Dual-Stream Attention Module (DSAM), which facilitates bidirectional interaction between multilingual semantic features and detailed visual representations, thereby synthesizing unified features robust to textual variations; and (2) a Visual-Aware Adapter (VAA), a parameter-efficient fine-tuning strategy that dynamically injects these fused visual cues into the frozen LLM backbone. This design enables the model to align the visual context with linguistic reasoning effectively while minimizing computational costs. Extensive experiments on eight tasks on three public benchmarks demonstrate that VaaWIT significantly outperforms state-of-the-art (SOTA) open-source baselines and achieves competitive performance against proprietary models. These results validate the efficacy of integrating fine-grained visual perception into LLMs for complex Web content analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
