我们介绍了 Instruction Document Visual Question Answering(iDocVQA)数据集和 Large Language Document(LLaDoc)模型,分别用于训练 Language-Vision(LV)模型以进行文档分析和文档图像预测。通常,用于 DocVQA 任务的深度神经网络是在缺乏指令的数据集上训练的。我们表明,使用遵循指令的数据集可以提高性能。我们使用近期最先进(SotA)的 Large Language and Vision Assistant(LLaVA)1.5 作为基础模型,比较了其在多个文档相关数据集上的性能。我们还使用基于轮询的对象探测评估(Polling-based Object Probing Evaluation,POPE)数据集评估了衍生模型在对象幻觉方面的性能。结果表明,指令微调的性能达到零样本性能的 11 倍至 32 倍,并且比非指令(传统任务)微调高出 0.1% 至 4.2%。尽管有所提升,这些性能仍不及人类表现(94.36%),表明仍有很大的改进空间。
英文摘要:
We introduce Instruction Document Visual Question Answering (iDocVQA) dataset and Large Language Document (LLaDoc) model, for training Language-Vision (LV) models for document analysis and predictions on document images, respectively. Usually, deep neural networks for the DocVQA task are trained on datasets lacking instructions. We show that using instruction-following datasets improves performance. We compare performance across document-related datasets using the recent state-of-the-art (SotA) Large Language and Vision Assistant (LLaVA)1.5 as the base model. We also evaluate the performance of the derived models for object hallucination using the Polling-based Object Probing Evaluation (POPE) dataset. The results show that instruction-tuning performance ranges from 11X to 32X of zero-shot performance and from 0.1% to 4.2% over non-instruction (traditional task) finetuning. Despite the gains, these still fall short of human performance (94.36%), implying there's much room for improvement.
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Meituan Group(美团集团)
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Alibaba Group(阿里巴巴集团)
An Empirical Study of Instruction-tuning Large Language Models in Chinese
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);APUS AiLMe Lab(APUS AiLMe实验室)
What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning
ShanghaiTech University(上海科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);Meituan(美团);Alibaba Group(阿里巴巴集团);The Hong Kong University of Science and Technology(香港科技大学)
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);University of Science and Technology Beijing(北京科技大学);Institute of Information Science, Beijing Jiaotong University(北京交通大学信息科学研究所)
Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models
Graduate School of Artificial Intelligence, POSTECH, Republic of Korea(POSTECH人工智能研究生院,韩国);Department of Computer Science and Engineering, POSTECH, Republic of Korea(POSTECH计算机科学与工程系,韩国)
ESAinsTOD: A Unified End-to-End Schema-Aware Instruction-Tuning Framework for Task-Oriented Dialog Modeling
Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology(社会科学与信息检索研究中心,哈尔滨工业大学);School of Computer Science and Engineering, Central South University(计算机科学与工程学院,中南大学)
Nilekani Centre at AI4Bharat(AI4Bharat的Nilekani中心);IIIT D&M Kancheepuram(坎奇普兰IIIT D&M);Flipkart;University of Surrey(萨里大学);A * STAR(新加坡科技研究局);IIT Madras(马德拉斯印度理工学院);NICT(日本国立信息通信技术研究所);IBM Research(IBM研究院);Microsoft(微软)
Benchmarking Complex Instruction-Following with Multiple Constraints Composition
Tsinghua University(清华大学);University of Electronic Science and Technology of China(电子科技大学);Zhipu AI(智谱AI);China University of Geosciences(中国地质大学);Central China Normal University(华中师范大学)
RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format
Institute of Image Processing and Pattern Recoginition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University(图像处理与模式识别研究所,自动化与智能感知学院,上海交通大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));MoE Key Laboratory of System Control and Information Processing (Shanghai)(系统控制与信息处理国家重点实验室(上海))