arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.17672cs.CVcs.GR
收藏

BlenderAlchemy:利用视觉-语言模型编辑三维图形

BlenderAlchemy: Editing 3D Graphics with Vision-Language Models

Stanford University(斯坦福大学)

Ian Huang, Guandao Yang, Leonidas Guibas

AI总结:

本文提出BlenderAlchemy系统,利用视觉-语言模型并结合图像生成模型的“想象”参考图像,自动搜索Blender设计动作空间,以完成程序化材质、几何体编辑及产品渲染光照调整等繁琐任务。

摘要:

图形设计在电影制作和游戏设计等多种应用中具有重要意义。为了创建高质量场景,设计师通常需要在Blender等软件中花费数小时,期间可能需要交错并重复执行数百次操作,例如连接材质节点。此外,略有不同的设计目标可能需要完全不同的操作序列,这使得自动化变得困难。在本文中,我们提出了一种系统,利用GPT-4V等视觉-语言模型(VLMs)智能搜索设计动作空间,以得到能够满足用户意图的结果。具体而言,我们设计了一个基于视觉的编辑生成器和状态评估器,二者协同工作以找到实现目标的正确动作序列。受人类设计过程中视觉想象作用的启发,我们利用图像生成模型生成的“想象”参考图像来补充VLMs的视觉推理能力,为抽象语言描述提供视觉基础。在本文中,我们提供了实证证据,表明我们的系统能够为诸如根据文本和/或参考图像编辑程序化材质和几何体,以及在复杂场景中为产品渲染调整光照配置等任务,生成简单但繁琐的Blender编辑序列。

英文摘要:

Graphics design is important for various applications, including movie production and game design. To create a high-quality scene, designers usually need to spend hours in software like Blender, in which they might need to interleave and repeat operations, such as connecting material nodes, hundreds of times. Moreover, slightly different design goals may require completely different sequences, making automation difficult. In this paper, we propose a system that leverages Vision-Language Models (VLMs), like GPT-4V, to intelligently search the design action space to arrive at an answer that can satisfy a user's intent. Specifically, we design a vision-based edit generator and state evaluator to work together to find the correct sequence of actions to achieve the goal. Inspired by the role of visual imagination in the human design process, we supplement the visual reasoning capabilities of VLMs with "imagined" reference images from image-generation models, providing visual grounding of abstract language descriptions. In this paper, we provide empirical evidence suggesting our system can produce simple but tedious Blender editing sequences for tasks such as editing procedural materials and geometry from text and/or reference images, as well as adjusting lighting configurations for product renderings in complex scenes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑