多模态深度研究员:从零开始生成文本-图表交织报告的代理框架
Multimodal DeepResearcher: Generating Text-Chart Interleaved Reports From Scratch with Agentic Framework
AI总结:
多模态深度研究员通过代理框架实现从零开始生成文本-图表交织报告,利用FDV结构化文本表示提升可视化生成质量。
摘要:
可视化在有效传达概念和信息方面起着关键作用。最近在推理和检索增强生成方面的进展使大语言模型(LLMs)能够进行深度研究并生成全面的报告。尽管取得了进展,现有的深度研究框架主要集中在生成纯文本内容,使得自动生成交织文本和可视化的任务仍然缺乏探索。这一新任务在设计信息丰富的可视化以及有效地将其与文本报告整合方面提出了关键挑战。为了解决这些挑战,我们提出了可视化形式描述(FDV),这是一种结构化的文本表示形式,使LLMs能够学习并生成多样化、高质量的可视化。在此基础上,我们引入了多模态深度研究员,一种代理框架,将任务分解为四个阶段:(1)研究,(2)示例报告文本化,(3)规划,(4)多模态报告生成。为了评估生成的多模态报告,我们开发了多模态报告基准(MultimodalReportBench),其中包含100种多样的主题作为输入,以及5个专门的指标。在不同模型和评估方法上的广泛实验证明了多模态深度研究员的有效性。值得注意的是,使用相同的Claude 3.7 Sonnet模型,多模态深度研究员在基准方法上实现了82%的整体胜率。
英文摘要:
Visualizations play a crucial part in effective communication of concepts and information. Recent advances in reasoning and retrieval augmented generation have enabled Large Language Models (LLMs) to perform deep research and generate comprehensive reports. Despite its progress, existing deep research frameworks primarily focus on generating text-only content, leaving the automated generation of interleaved texts and visualizations underexplored. This novel task poses key challenges in designing informative visualizations and effectively integrating them with text reports. To address these challenges, we propose Formal Description of Visualization (FDV), a structured textual representation of charts that enables LLMs to learn from and generate diverse, high-quality visualizations. Building on this representation, we introduce Multimodal DeepResearcher, an agentic framework that decomposes the task into four stages: (1) researching, (2) exemplar report textualization, (3) planning, and (4) multimodal report generation. For the evaluation of generated multimodal reports, we develop MultimodalReportBench, which contains 100 diverse topics served as inputs along with 5 dedicated metrics. Extensive experiments across models and evaluation methods demonstrate the effectiveness of Multimodal DeepResearcher. Notably, utilizing the same Claude 3.7 Sonnet model, Multimodal DeepResearcher achieves an 82% overall win rate over the baseline method.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。