arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.00453cs.CVcs.CL
收藏

指令带来差异

Instruction Makes a Difference

Luleå University of Technology(吕勒奥理工大学)

Tosin Adewumi, Nudrat Habib, Lama Alkhaled, Elisa Barney

AI总结:

该研究提出 iDocVQA 数据集与 LLaDoc 模型,证明指令微调可显著提升 LLaVA-1.5 在文档视觉问答及对象幻觉评估中的表现,但与人类水平仍有差距。

摘要:

我们介绍了 Instruction Document Visual Question Answering(iDocVQA)数据集和 Large Language Document(LLaDoc)模型,分别用于训练 Language-Vision(LV)模型以进行文档分析和文档图像预测。通常,用于 DocVQA 任务的深度神经网络是在缺乏指令的数据集上训练的。我们表明,使用遵循指令的数据集可以提高性能。我们使用近期最先进(SotA)的 Large Language and Vision Assistant(LLaVA)1.5 作为基础模型,比较了其在多个文档相关数据集上的性能。我们还使用基于轮询的对象探测评估(Polling-based Object Probing Evaluation,POPE)数据集评估了衍生模型在对象幻觉方面的性能。结果表明,指令微调的性能达到零样本性能的 11 倍至 32 倍,并且比非指令(传统任务)微调高出 0.1% 至 4.2%。尽管有所提升,这些性能仍不及人类表现(94.36%),表明仍有很大的改进空间。

英文摘要:

We introduce Instruction Document Visual Question Answering (iDocVQA) dataset and Large Language Document (LLaDoc) model, for training Language-Vision (LV) models for document analysis and predictions on document images, respectively. Usually, deep neural networks for the DocVQA task are trained on datasets lacking instructions. We show that using instruction-following datasets improves performance. We compare performance across document-related datasets using the recent state-of-the-art (SotA) Large Language and Vision Assistant (LLaVA)1.5 as the base model. We also evaluate the performance of the derived models for object hallucination using the Polling-based Object Probing Evaluation (POPE) dataset. The results show that instruction-tuning performance ranges from 11X to 32X of zero-shot performance and from 0.1% to 4.2% over non-instruction (traditional task) finetuning. Despite the gains, these still fall short of human performance (94.36%), implying there's much room for improvement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑