视觉大模型 / VLM - arXivDaily 专题

2501.16664 2025-01-29 cs.RO cs.CV cs.LG 62%

Improving Vision-Language-Action Model with Online Reinforcement Learning

Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, Jianyu Chen

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted to ICRA 2025

2501.12326 2025-01-22 cs.AI cs.CL cs.CV cs.HC 62%

UI-TARS: Pioneering Automated GUI Interaction with Native Agents

Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, Guang Shi

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV、cs.AI

2501.09355 2025-01-17 cs.AI cs.CV cs.ET cs.MA 62%

YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks

Saptarashmi Bandyopadhyay, Vikas Bahirwani, Lavisha Aggarwal, Bhanu Guda, Lin Li, Andrea Colaco

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Preprint

2412.11484 2024-12-17 cs.AI cs.CV cs.RO 62%

Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents

Wonje Choi, Woo Kyung Kim, SeungHyun Kim, Honguk Woo

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at NeurIPS 2023

2411.05821 2024-12-10 cs.RO cs.CV cs.LG 62%

Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks

Pranav Guruprasad, Harshvardhan Sikka, Jaewoo Song, Yangyue Wang, Paul Pu Liang

专题命中 GUI与屏幕智能体：VLM(abstract)；分类 cs.CV、cs.LG

Comments 16 Pages, 10 Figures

2411.17465 2024-11-27 cs.CV cs.AI cs.CL cs.HC 62%

ShowUI: One Vision-Language-Action Model for GUI Visual Agent

Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, Mike Zheng Shou

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV、cs.AI

Comments Technical Report. Github: https://github.com/showlab/ShowUI

2411.13451 2024-11-21 cs.AI cs.CL cs.LG 62%

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso

专题命中 GUI与屏幕智能体：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments 18 pages, 3 figures, an abridged version to appear in NeurIPS 2024 AFM Workshop

2410.18481 2024-11-20 cs.CL cs.AI cs.LG 62%

Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction

Sergio Burdisso, Srikanth Madikeri, Petr Motlicek

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.AI、cs.LG

Comments Accepted to EMNLP 2024 main conference

Journal ref https://aclanthology.org/2024.emnlp-main.310/

2411.00820 2024-11-05 cs.HC cs.AI cs.CL cs.LG 62%

AutoGLM: Autonomous Foundation Agents for GUIs

Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, Junjie Gao, Junjun Shan, Kangning Liu, Shudan Zhang, Shuntian Yao, Siyi Cheng, Wentao Yao, Wenyi Zhao, Xinghan Liu, Xinyi Liu, Xinying Chen, Xinyue Yang, Yang Yang, Yifan Xu, Yu Yang, Yujia Wang, Yulin Xu, Zehan Qi, Yuxiao Dong, Jie Tang

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.AI、cs.LG

2410.23254 2024-10-31 cs.RO cs.AI cs.CV 62%

Keypoint Abstraction using Large Models for Object-Relative Imitation Learning

Xiaolin Fang, Bo-Ruei Huang, Jiayuan Mao, Jasmine Shone, Joshua B. Tenenbaum, Tomás Lozano-Pérez, Leslie Pack Kaelbling

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments CoRL LangRob Workshop, 2024

2409.04081 2024-10-03 cs.CL cs.AI cs.HC cs.LG 62%

UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity

Yicheng Fu, Raviteja Anantha, Prabal Vashisht, Jianpeng Cheng, Etai Littwin

专题命中 GUI与屏幕智能体：multimodal large language model(abstract)；分类 cs.AI、cs.LG

2406.17768 2024-09-20 cs.RO cs.AI cs.LG 62%

EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data

Jesse Zhang, Minho Heo, Zuxin Liu, Erdem Biyik, Joseph J Lim, Yao Liu, Rasool Fakoor

专题命中 GUI与屏幕智能体：vision language model(abstract)；分类 cs.AI、cs.LG

Comments 25 pages, 16 figures

Journal ref CoRL 2024

2408.08282 2024-08-16 cs.RO cs.AI cs.LG 62%

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Jin Wang, Arturo Laurenzi, Nikos Tsagarakis

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.AI、cs.LG

Comments Paper accepted by IROS 2024

2407.20798 2024-07-31 cs.LG cs.AI cs.RO 62%

Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

Norman Di Palo, Leonard Hasenclever, Jan Humplik, Arunkumar Byravan

专题命中 GUI与屏幕智能体：vision language model(abstract)；分类 cs.AI、cs.LG

Comments Published at 3rd Conference on Lifelong Learning Agents (CoLLAs), 2024

2406.11815 2024-06-18 cs.RO cs.CV cs.LG 62%

LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

Dantong Niu, Yuvan Sharma, Giscard Biamby, Jerome Quenum, Yutong Bai, Baifeng Shi, Trevor Darrell, Roei Herzig

专题命中 GUI与屏幕智能体：visual question answering(abstract)；分类 cs.CV、cs.LG

2310.18297 2024-02-23 cs.CV cs.AI 62%

Image Clustering Conditioned on Text Criteria

Sehyun Kwon, Jaeseung Park, Minkyu Kim, Jaewoong Cho, Ernest K. Ryu, Kangwook Lee

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV、cs.AI

2311.14656 2024-01-17 cs.CV cs.AI 62%

Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs

Jonathan Roberts, Timo Lüddecke, Rehan Sheikh, Kai Han, Samuel Albanie

专题命中 GUI与屏幕智能体：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments V3: Fixed typo in Fig.1; V2: Minor formatting changes and added missing subfigure captions

2310.15127 2023-11-21 cs.AI cs.CL cs.LG cs.RO 62%

Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models

Gabriel Sarch, Yue Wu, Michael J. Tarr, Katerina Fragkiadaki

专题命中 GUI与屏幕智能体：VLM(abstract)；分类 cs.AI、cs.LG

Comments Project page with code & videos: https://helper-agent-llm.github.io

2309.14774 2023-09-27 cs.LG cs.CL cs.CV cs.HC 62%

BLIP-Adapter: Parameter-Efficient Transfer Learning for Mobile Screenshot Captioning

Ching-Yu Chiang, I-Hua Chang, Shih-Wei Liao

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV、cs.LG

2210.03112 2023-04-18 cs.LG cs.CL cs.CV cs.RO 62%

A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning

Aishwarya Kamath, Peter Anderson, Su Wang, Jing Yu Koh, Alexander Ku, Austin Waters, Yinfei Yang, Jason Baldridge, Zarana Parekh

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV、cs.LG

Comments CVPR 2023

2207.06415 2022-07-15 cs.LG cs.AI q-bio.NC 62%

The Free Energy Principle for Perception and Action: A Deep Learning Perspective

Pietro Mazzaglia, Tim Verbelen, Ozan Çatal, Bart Dhoedt

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.AI、cs.LG

Journal ref Entropy 2022

2008.01156 2021-02-08 cs.RO cs.AI cs.CV 62%

Action sequencing using visual permutations

Michael Burke, Kartic Subr, Subramanian Ramamoorthy

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV、cs.AI

Comments This paper has been accepted for publication at IEEE RA-L

2003.03220 2020-03-09 cs.AI cs.LG 62%

Deep Active Inference for Autonomous Robot Navigation

Ozan Çatal, Samuel Wauthier, Tim Verbelen, Cedric De Boom, Bart Dhoedt

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.AI、cs.LG

Comments workshop paper at BAICS at ICLR 2020

1811.10092 2019-04-09 cs.CV cs.AI cs.CL cs.RO 62%

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2019 Oral

1901.03035 2019-01-11 cs.AI cs.CL cs.CV cs.RO 62%

Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, Caiming Xiong

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV、cs.AI

Comments ICLR 2019, code is available at https://github.com/chihyaoma/selfmonitoring-agent

1803.07729 2018-07-27 cs.CV cs.AI cs.CL cs.RO 62%

Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation

Xin Wang, Wenhan Xiong, Hongmin Wang, William Yang Wang

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV、cs.AI

Comments 21 pages, 7 figures, with supplementary material

1711.07280 2018-04-09 cs.CV cs.AI cs.CL cs.RO 62%

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, Anton van den Hengel

专题命中 GUI与屏幕智能体：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2018 Spotlight presentation

1707.08668 2017-07-28 cs.AI cs.CL 61%

A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions

Siddharth Karamcheti, Edward C. Williams, Dilip Arumugam, Mina Rhee, Nakul Gopalan, Lawson L. S. Wong, Stefanie Tellex

专题命中 GUI与屏幕智能体：grounding(abstract,comments)；分类 cs.AI

Comments Accepted at the 1st Workshop on Language Grounding for Robotics at ACL 2017

2608.21247 2026-08-24 cs.CV cs.RO 新提交 57%

Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models

视觉-语言-动作模型中用于令牌压缩的刚可察觉差异建模

Zhuoyuan Li, Rui Zhao, Jin Wang, Hanwei Zhu, Cong Zhang, Giuseppe Valenzise, Weisi Lin, Kin-Man Lam

机构 * The Hong Kong Polytechnic University（香港理工大学）； Nanyang Technological University（南洋理工大学）； Bytedance Inc.（字节跳动公司）； Shandong University（山东大学）； CNRS（法国国家科学研究中心）； CentraleSupelec（中央理工学院）； Université Paris-Saclay（巴黎萨克雷大学）

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

AI总结本文针对视觉-语言-动作模型的令牌压缩问题，提出Action-JND方法，通过动作容忍度准则优化压缩，在LIBERO基准实验中提升了激进压缩下的压缩可靠性。

Comments 15 pages, 5 figures

2607.28645 2026-08-24 cs.HC cs.AI 版本更新 57%

Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation

看起来对，运行起来对：面向多屏移动应用生成的项目级基准测试

Fan Wu, Cuiyun Gao, Yiming Huang, Yang Xiao, Yujia Chen, Qing Liao

专题命中 GUI与屏幕智能体：multimodal large language model(abstract)；分类 cs.AI

AI总结针对现有设计转代码基准的局限，提出首个项目级多屏移动应用生成基准MobileForge，通过五轴评估及两种测试方法，发现前沿多模态LLM构建的应用存在导航、保真度和可维护性问题。

Comments Accepted by EMNLP 2026 Main

1. GUI与屏幕智能体 1576 篇

Improving Vision-Language-Action Model with Online Reinforcement Learning

UI-TARS: Pioneering Automated GUI Interaction with Native Agents

YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks

Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents

Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks

ShowUI: One Vision-Language-Action Model for GUI Visual Agent

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction

AutoGLM: Autonomous Foundation Agents for GUIs

Keypoint Abstraction using Large Models for Object-Relative Imitation Learning

UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity

EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data

Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model

Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learning

LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning

Image Clustering Conditioned on Text Criteria

Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs

Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models

BLIP-Adapter: Parameter-Efficient Transfer Learning for Mobile Screenshot Captioning

A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learning

The Free Energy Principle for Perception and Action: A Deep Learning Perspective

Action sequencing using visual permutations

Deep Active Inference for Autonomous Robot Navigation

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Self-Monitoring Navigation Agent via Auxiliary Progress Estimation

Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation

Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments

A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructions

Just Noticeable Difference Modeling for Token Compression in Vision-Language-Action Models

Looks Right, Works Right: A Project-Level Benchmark for Multi-Screen Mobile App Generation