GPT-4V(ision) 是一个通用型网页智能体，若具备接地能力
GPT-4V(ision) is a Generalist Web Agent, if Grounded
AI总结：
本文提出SEEACT，利用GPT-4V作为通用网页智能体，通过视觉与HTML结合实现接地，在MIND2WEB基准上显著优于文本模型，但接地仍是主要挑战。
摘要：
大型多模态模型（LMMs）的最新发展，特别是GPT-4V(ision)和Gemini，正迅速扩展多模态模型的能力边界，超越图像描述和视觉问答等传统任务。在本工作中，我们探索了如GPT-4V等大型多模态模型作为通用型网页智能体的潜力，该智能体能够遵循自然语言指令在任意给定网站上完成任务。我们提出了SEEACT，一个通用型网页智能体，利用大型多模态模型的能力进行集成的视觉理解和网页操作。我们在最新的MIND2WEB基准上进行了评估。除了在缓存网站上进行标准的离线评估外，我们还通过开发一个工具，使得网页智能体能够在实时网站上运行，从而启用了一种新的在线评估设置。我们表明，GPT-4V为网页智能体展现了巨大的潜力——如果我们将它的文本计划手动接地为网站上的操作，它能够成功完成实时网站上51.1%的任务。这显著优于仅文本的大型语言模型（如GPT-4）或专门为网页智能体微调的较小模型（FLAN-T5和BLIP-2）。然而，接地仍然是一个主要挑战。现有的LMM接地策略，如set-of-mark提示，对网页智能体并不有效，而我们在本文中开发的最佳接地策略利用了HTML结构和视觉信息。尽管如此，与神谕接地相比仍有显著差距，留下了充足的进一步改进空间。所有代码、数据和评估工具可在 https://github.com/OSU-NLP-Group/SeeAct 获取。
英文摘要：
The recent development on large multimodal models (LMMs), especially GPT-4V(ision) and Gemini, has been quickly expanding the capability boundaries of multimodal models beyond traditional tasks like image captioning and visual question answering. In this work, we explore the potential of LMMs like GPT-4V as a generalist web agent that can follow natural language instructions to complete tasks on any given website. We propose SEEACT, a generalist web agent that harnesses the power of LMMs for integrated visual understanding and acting on the web. We evaluate on the recent MIND2WEB benchmark. In addition to standard offline evaluation on cached websites, we enable a new online evaluation setting by developing a tool that allows running web agents on live websites. We show that GPT-4V presents a great potential for web agents -- it can successfully complete 51.1 of the tasks on live websites if we manually ground its textual plans into actions on the websites. This substantially outperforms text-only LLMs like GPT-4 or smaller models (FLAN-T5 and BLIP-2) specifically fine-tuned for web agents. However, grounding still remains a major challenge. Existing LMM grounding strategies like set-of-mark prompting turns out to be not effective for web agents, and the best grounding strategy we develop in this paper leverages both the HTML structure and visuals. Yet, there is still a substantial gap with oracle grounding, leaving ample room for further improvement. All code, data, and evaluation tools are available at https://github.com/OSU-NLP-Group/SeeAct.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
