基于指令遵循模型的视觉丰富文档自动布局规划
Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models
AI总结：
本文提出一种用于布局规划的多模态指令遵循框架，通过三个布局推理任务训练模型，使非专业用户能便捷创建定制布局，在 Crello 上 mIoU 比 GPT-4V 高 12%。
摘要：
近期指令遵循模型的进展使用户与模型的交互更加友好高效，拓宽了其适用性。在平面设计中，非专业用户常因技能和资源有限而难以创建视觉美观的布局。在本工作中，我们提出了一种新颖的用于布局规划的多模态指令遵循框架，允许用户通过指定画布大小和设计目的（如用于书籍封面、海报、宣传册或菜单），轻松将视觉元素排列成定制布局。我们开发了三个布局推理任务来训练模型理解和执行布局指令。在两个基准上的实验表明，我们的方法不仅简化了非专业用户的设计流程，还超越了少样本 GPT-4V 模型的性能，在 Crello 上 mIoU 高出 12%。这一进展凸显了多模态指令遵循模型在自动化和简化设计流程方面的潜力，为视觉丰富文档上的各种设计任务提供了易于使用的解决方案。
英文摘要：
Recent advancements in instruction-following models have made user interactions with models more user-friendly and efficient, broadening their applicability. In graphic design, non-professional users often struggle to create visually appealing layouts due to limited skills and resources. In this work, we introduce a novel multimodal instruction-following framework for layout planning, allowing users to easily arrange visual elements into tailored layouts by specifying canvas size and design purpose, such as for book covers, posters, brochures, or menus. We developed three layout reasoning tasks to train the model in understanding and executing layout instructions. Experiments on two benchmarks show that our method not only simplifies the design process for non-professionals but also surpasses the performance of few-shot GPT-4V models, with mIoU higher by 12% on Crello. This progress highlights the potential of multimodal instruction-following models to automate and simplify the design process, providing an approachable solution for a wide range of design tasks on visually-rich documents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
