arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.04183cs.CL
收藏

OpenViVQA:面向越南语视觉问答的任务、数据集与多模态融合模型

OpenViVQA: Task, Dataset, and Multimodal Fusion Models for Visual Question Answering in Vietnamese

University of Information Technology(信息科技大学);Vietnam National University, Ho Chi Minh city(胡志明市越南国家大学)

Nghia Hieu Nguyen, Duong T. D. Vo, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

AI总结:

针对现有VQA数据集多面向高资源语言、且多为答案选择任务的局限,本文发布首个越南语开放式答案大规模VQA数据集OpenViVQA,并提出三种多模态融合模型,效果与主流SOTA模型相当,助力低资源语言VQA研究。

摘要:

近年来,视觉问答(VQA)凭借其高潜力的应用场景(如智能汽车虚拟助手、盲人辅助设备、以自然语言为查询从文档图像中检索信息等)以及自身的挑战性,受到了研究界的广泛关注。VQA任务要求相关方法具备融合问题与图像信息以生成恰当答案的能力。神经视觉问答模型在大规模数据集上已取得巨大进展,但这些数据集大多面向英语等资源丰富的语言。此外,现有数据集将VQA任务限定为答案选择任务或答案分类任务。我们认为这种形式的VQA远未达到人类能力水平,它仅通过选择答案而非生成答案的方式,消解了VQA任务中回答环节的挑战性。本文提出了OpenViVQA(开放域越南语视觉问答)数据集,这是首个面向越南语的开放式答案大规模VQA数据集,包含11000余张图像,对应37000余组问答对(QA)。此外,我们提出了FST、QuMLAG和MLPAG模型,这些模型融合图像与问题的信息,再利用融合后的特征以人类迭代思考的方式构建答案。我们提出的方法取得了与SAAA、MCAN、LORA、M4C等SOTA模型相当的结果。该数据集已开放获取,旨在鼓励研究界为越南语等低资源语言开发更具泛化性的算法,包括基于transformer的算法。

英文摘要:

In recent years, visual question answering (VQA) has attracted attention from the research community because of its highly potential applications (such as virtual assistance on intelligent cars, assistant devices for blind people, or information retrieval from document images using natural language as queries) and challenge. The VQA task requires methods that have the ability to fuse the information from questions and images to produce appropriate answers. Neural visual question answering models have achieved tremendous growth on large-scale datasets which are mostly for resource-rich languages such as English. However, available datasets narrow the VQA task as the answers selection task or answer classification task. We argue that this form of VQA is far from human ability and eliminates the challenge of the answering aspect in the VQA task by just selecting answers rather than generating them. In this paper, we introduce the OpenViVQA (Open-domain Vietnamese Visual Question Answering) dataset, the first large-scale dataset for VQA with open-ended answers in Vietnamese, consists of 11,000+ images associated with 37,000+ question-answer pairs (QAs). Moreover, we proposed FST, QuMLAG, and MLPAG which fuse information from images and answers, then use these fused features to construct answers as humans iteratively. Our proposed methods achieve results that are competitive with SOTA models such as SAAA, MCAN, LORA, and M4C. The dataset is available to encourage the research community to develop more generalized algorithms including transformers for low-resource languages such as Vietnamese.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑