Multi-Agent Interactive Question Generation Framework for Long Document Understanding
Kesen Wang, Daulet Toibazar, Abdulrahman Alfulayt, Abdulaziz S. Albadawi, Ranya A. Alkahtani, Asma A. Ibrahim, Haneen A. Alhomoud, Sherif Mohamed, Pedro J. Moreno
在具有复杂版式的长上下文场景中,文档理解(Document Understanding,DU)仍是视觉-语言研究中的一项重大挑战。尽管大型视觉-语言模型(Large Vision-Language Models,LVLMs)擅长短上下文 DU 任务,但它们在长上下文设置下的性能会下降。一个关键限制是细粒度训练数据的稀缺,尤其是在阿拉伯语等低资源语言方面。现有最先进技术严重依赖人工标注,成本高昂且效率低下。我们提出了一种全自动的多智能体交互式框架,以高效生成长上下文问题。我们的方法能够针对涵盖数百页、跨越多个不同领域的大量英文和阿拉伯文文档,高效生成高质量的单页和多页问题。这促进了具有更强长上下文理解能力的 LVLMs 的开发。本文的实验结果表明,我们生成的英文和阿拉伯文问题(AraEngLongBench)对主流开源和闭源 LVLMs 都具有相当大的挑战性。本文提出的代码和数据可在 https://github.com/wangk0b/Multi_Agentic_QA_Long_Doc.git 获取。示例问答(Question and Answer,QA)对和结构化系统提示可在附录中找到。
英文摘要:
Document Understanding (DU) in long-contextual scenarios with complex layouts remains a significant challenge in vision-language research. Although Large Vision-Language Models (LVLMs) excel at short-context DU tasks, their performance declines in long-context settings. A key limitation is the scarcity of fine-grained training data, particularly for low-resource languages such as Arabic. Existing state-of-the-art techniques rely heavily on human annotation, which is costly and inefficient. We propose a fully automated, multi-agent interactive framework to generate long-context questions efficiently. Our approach efficiently generates high-quality single- and multi-page questions for extensive English and Arabic documents, covering hundreds of pages across diverse domains. This facilitates the development of LVLMs with enhanced long-context understanding ability. Experimental results in this work have shown that our generated English and Arabic questions (\textbf{AraEngLongBench}) are quite challenging to major open- and close-source LVLMs. The code and data proposed in this work can be found in https://github.com/wangk0b/Multi_Agentic_QA_Long_Doc.git. Sample Question and Answer (QA) pairs and structured system prompts can be found in the Appendix.
Visual Document Understanding and Reasoning: A Multi-Agent Collaboration Framework with Agent-Wise Adaptive Test-Time Scaling
National University of Singapore(新加坡国立大学);Tencent Youtu Lab(腾讯优图实验室);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学);Nanyang Technological University(南洋理工大学);Zhejiang University(浙江大学);DeepWisdom(深智科技)
Aligning to Social Norms and Values in Interactive Narratives
Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学)
我们专注于创建代理,在互动叙事或基于文本的游戏中与社会有益的规范和价值观保持一致-环境中,代理通过自然语言感知并与世界互动。这种交互式代理通常通过强化学习进行训练,以优化任务性能,即使这种奖励可能导致违反社会规范的代理行为-对代理本身或环境中的其他实体造成伤害。社会价值对齐是指创造行为符合特定背景和人群预期道德和社会规范的代理人-在我们的情况下,它意味着代理人的行为方式对自己和他人的危害较小,更有益。我们建立在Jiminy Cricket基准(Hendrycks et al. 2021)的基础上,这是一组25个注释的交互式叙述,包含数千个道德上突出的场景,涵盖了从盗窃和身体伤害到利他主义的一切。我们引入GALAD(游戏-价值对齐通过行动蒸馏)代理,它使用专门训练的语言模型中存在的社会常识知识,根据上下文将其动作空间限制为仅与社会有益价值对齐的那些动作。一项实验研究表明,GALAD代理有效地做出决策,足以将最先进的任务性能提高4%,同时与强大的当代价值对齐方法相比,将社会有害行为的频率降低25%。
IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Massachusetts Institute of Technology(麻省理工学院);University of Waterloo(滑铁卢大学);Meta AI;AIRI(人工智能研究院);École Polytechnique de Montréal(蒙特利尔理工学院);Microsoft Research(微软研究院)
Knowledge-enhanced Agents for Interactive Text Games
Information Sciences Institute, University of Southern California(南加州大学信息科学研究所);Bosch Center for Artificial Intelligence(博世人工智能中心);Carnegie Mellon University(卡内基梅隆大学)
Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件工程学院);MiLM Plus, Xiaomi Inc.(小米公司MiLM Plus团队);School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of Data Science, Fudan University(复旦大学数据科学学院)
DashboardQA: Benchmarking Multimodal Agents for Question Answering on Interactive Dashboards
York University(约克大学);RBC;Qatar Computing Research Institute (QCRI)(卡塔尔计算研究所);Nanyang Technological University(南洋理工大学);Salesforce Research(Salesforce研究)
Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalization
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国);Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China(计算机科学与技术系,人工智能研究院,清华大学,北京,中国);Hong Kong University of Science and Technology, Hong Kong, China(香港科技大学,香港,中国);Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团)