arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00393cs.CVcs.CLcs.LG
收藏

巴西葡萄牙语图像描述生成:跨原生翻译数据集研究

Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset

Instituto Federal do Espírito Santo (IFES)(巴西联邦大学埃斯皮里图圣安东尼奥理工学院);École de Technologie Supérieure (ÉTS)(加拿大超技术学院)

Gabriel Bromonschenkel, Alessandro L. Koerich, Thiago M. Paixão, Hilário Tomaz Alves de Oliveira

AI总结:

本研究通过跨原生翻译数据集评估Transformer模型在巴西葡萄牙语图像描述生成中的表现,发现Swin-DistilBERTimbau表现优异,GPT-4在CLIP-Score上表现最佳,同时揭示了模型中的系统性偏见。

摘要:

图像描述生成(IC)指的是自动为图像生成自然语言描述,其应用范围从社交媒体内容生成到帮助视障人士。尽管大多数研究集中在英语模型上,但低资源语言如巴西葡萄牙语面临显著挑战,因为缺乏专门的数据集和模型。一些研究通过自动翻译现有数据集来缓解资源稀缺问题。本研究通过提出基于Transformer的视觉和语言模型在巴西葡萄牙语图像描述生成中的跨原生翻译评估来填补这一空白。我们使用包含由原生巴西葡萄牙语说话者手动创建的描述的Flickr30K版本,并将其与通过将英语自动翻译成葡萄牙语生成的版本进行比较。实验包括一种跨上下文方法,即在一种数据集上训练的模型在另一种数据集上进行测试以评估翻译影响。此外,我们还整合了注意力图用于模型推理解释,并使用CLIP-Score指标来评估图像-描述对齐度。我们的发现表明,Swin-DistilBERTimbau在跨数据集上表现优异,显示出强大的泛化能力。ViTucano,一种预训练的巴西葡萄牙语VLM,在传统文本评估指标上超过了更大的多语言模型(GPT-4o、LLaMa 3.2 Vision),而GPT-4模型在CLIP-Score上取得最高分,表明改进的图像-文本对齐。注意力分析揭示了系统性的偏见,包括性别误分类、物体计数错误和空间不一致。本研究中生成和分析的数据集和模型可在:https://github.com/laicsiifes/transformer-caption-ptbr上找到。

英文摘要:

Image captioning (IC) refers to the automatic generation of natural language descriptions for images, with applications ranging from social media content generation to assisting individuals with visual impairments. While most research has been focused on English-based models, low-resource languages such as Brazilian Portuguese face significant challenges due to the lack of specialized datasets and models. Several studies create datasets by automatically translating existing ones to mitigate resource scarcity. This work addresses this gap by proposing a cross-native-translated evaluation of Transformer-based vision and language models for Brazilian Portuguese IC. We use a version of Flickr30K comprised of captions manually created by native Brazilian Portuguese speakers and compare it to a version with captions automatically translated from English to Portuguese. The experiments include a cross-context approach, where models trained on one dataset are tested on the other to assess the translation impact. Additionally, we incorporate attention maps for model inference interpretation and use the CLIP-Score metric to evaluate the image-description alignment. Our findings show that Swin-DistilBERTimbau consistently outperforms other models, demonstrating strong generalization across datasets. ViTucano, a Brazilian Portuguese pre-trained VLM, surpasses larger multilingual models (GPT-4o, LLaMa 3.2 Vision) in traditional text-based evaluation metrics, while GPT-4 models achieve the highest CLIP-Score, highlighting improved image-text alignment. Attention analysis reveals systematic biases, including gender misclassification, object enumeration errors, and spatial inconsistencies. The datasets and the models generated and analyzed during the current study are available in: https://github.com/laicsiifes/transformer-caption-ptbr.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑