视觉大模型 / VLM - arXivDaily 专题

2411.10234 2024-11-18 cs.HC cs.AI 57%

Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability

J. Bieniek, M. Rahouti, D. C. Verma

专题命中 GUI与屏幕智能体：multimodal large language model(abstract)；分类 cs.AI

Comments 13 pages, 4 figures

2411.08579 2024-11-14 cs.CV cs.RO 57%

NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation

Youzhi Liu, Fanglong Yao, Yuanchang Yue, Guangluan Xu, Xian Sun, Kun Fu

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

2404.00318 2024-11-06 cs.RO cs.CV 57%

Cognitive Planning for Object Goal Navigation using Generative AI Models

Arjun P S, Andrew Melnik, Gora Chand Nandi

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

2410.19461 2024-11-05 cs.AI 57%

EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

Xuetian Chen, Hangcheng Li, Jiaqing Liang, Sihang Jiang, Deqing Yang

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.AI

2410.21926 2024-10-30 cs.RO cs.AI 57%

Reliable Semantic Understanding for Real World Zero-shot Object Goal Navigation

Halil Utku Unlu, Shuaihang Yuan, Congcong Wen, Hao Huang, Anthony Tzes, Yi Fang

专题命中 GUI与屏幕智能体：vision language model(abstract)；分类 cs.AI

Comments 16 pages, 7 figures, 2 tables

2408.04168 2024-10-18 cs.AI 57%

Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, Yong Li

专题命中 GUI与屏幕智能体：LLaVA(abstract)；分类 cs.AI

2407.13505 2024-10-14 cs.RO cs.AI 57%

Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation

Hassan Ali, Philipp Allgeuer, Carlo Mazzola, Giulia Belgiovine, Burak Can Kaplan, Lukáš Gajdošech, Stefan Wermter

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.AI

2410.07087 2024-10-11 cs.CV cs.RO 57%

Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology

Xiangyu Wang, Donglin Yang, Ziqin Wang, Hohin Kwan, Jinyu Chen, Wenjun Wu, Hongsheng Li, Yue Liao, Si Liu

专题命中 GUI与屏幕智能体：MLLM(abstract)；分类 cs.CV

2407.03000 2024-10-11 cs.CL cs.CV 57%

VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values

Zhe Hu, Yixiao Ren, Jing Li, Yu Yin

专题命中 GUI与屏幕智能体：vision language model(abstract)；分类 cs.CV

Comments EMNLP 2024 Main Conference

2410.05191 2024-10-08 cs.RO cs.AI 57%

LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulation

Zhijie Wang, Zhehua Zhou, Jiayang Song, Yuheng Huang, Zhan Shu, Lei Ma

专题命中 GUI与屏幕智能体：vision language model(abstract)；分类 cs.AI

Comments 8 pages, 4 figures

2408.01877 2024-10-03 cs.RO cs.CV 57%

Improving Zero-Shot ObjectNav with Generative Communication

Vishnu Sashank Dorbala, Vishnu Dutt Sharma, Pratap Tokekar, Dinesh Manocha

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

2409.15806 2024-09-25 cs.AI 57%

CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation

Fuxian Huang, Qi Zhang, Shaopeng Zhai, Jie Wang, Tianyi Zhang, Haoran Zhang, Ming Zhou, Yu Liu, Yu Qiao

专题命中 GUI与屏幕智能体：multimodal large language model(abstract)；分类 cs.AI

2409.12889 2024-09-24 cs.AI 57%

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

Peng Chen, Pi Bu, Jun Song, Yuan Gao, Bo Zheng

专题命中 GUI与屏幕智能体：vision language model(abstract)；分类 cs.AI

2409.08444 2024-09-16 cs.CV 57%

Towards Unified Facial Action Unit Recognition Framework by Large Language Models

Guohong Hu, Xing Lan, Hanyu Jiang, Jiayi Lyu, Jian Xue

专题命中 GUI与屏幕智能体：LLaVA(abstract)；分类 cs.CV

2406.18977 2024-09-13 cs.RO cs.CL cs.CV 57%

RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation

Fanfan Liu, Feng Yan, Liming Zheng, Chengjian Feng, Yiyang Huang, Lin Ma

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

2408.16412 2024-09-12 cs.CV 57%

Text-Enhanced Zero-Shot Action Recognition: A training-free approach

Massimo Bosetti, Shibingfeng Zhang, Benedetta Liberatori, Giacomo Zara, Elisa Ricci, Paolo Rota

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

Comments accepted to ICPR 2024

2409.05865 2024-09-10 cs.RO cs.LG 57%

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments

Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah

专题命中 GUI与屏幕智能体：MLLM(abstract)；分类 cs.LG

Comments Project website https://robotutilitymodels.com

2406.09246 2024-09-09 cs.RO cs.LG 57%

OpenVLA: An Open-Source Vision-Language-Action Model

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, Chelsea Finn

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.LG

Comments Website: https://openvla.github.io/

2407.15706 2024-08-16 cs.CV 57%

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

Jinfu Liu, Chen Chen, Mengyuan Liu

专题命中 GUI与屏幕智能体：multimodal large language model(abstract)；分类 cs.CV

2408.05090 2024-08-12 cs.CV cs.MM 57%

Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigation

Huilin Tian, Jingke Meng, Wei-Shi Zheng, Yuan-Ming Li, Junkai Yan, Yunong Zhang

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV

Comments arXiv admin note: text overlap with arXiv:2203.13838 by other authors

2407.09503 2024-07-26 cs.CV cs.HC cs.NE 57%

PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos

Steven Abreu, Tiffany D. Do, Karan Ahuja, Eric J. Gonzalez, Lee Payne, Daniel McDuff, Mar Gonzalez-Franco

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV

2311.17944 2024-07-19 cs.CV 57%

PALM: Predicting Actions through Language Models

Sanghwan Kim, Daoji Huang, Yongqin Xian, Otmar Hilliges, Luc Van Gool, Xi Wang

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

2403.11650 2024-07-18 cs.CV 57%

Prioritized Semantic Learning for Zero-shot Instance Navigation

Xinyu Sun, Lizhao Liu, Hongyan Zhi, Ronghe Qiu, Junwei Liang

专题命中 GUI与屏幕智能体：vision-language model(abstract)；分类 cs.CV

Comments Accepted by ECCV 2024

2407.10528 2024-07-16 cs.CV 57%

Local Action-Guided Motion Diffusion Model for Text-to-Motion Generation

Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Runyi Yu, Chang Liu, Xiangyang Ji, Li Yuan, Jie Chen

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.CV

Comments Accepted by ECCV 2024

2407.03913 2024-07-08 cs.AI cs.HC 57%

MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, Jianping Fan

专题命中 GUI与屏幕智能体：visual language model(abstract)；分类 cs.AI

2406.17520 2024-06-26 cs.CV cs.RO 57%

Tell Me Where You Are: Multimodal LLMs Meet Place Recognition

Zonglin Lyu, Juexiao Zhang, Mingxuan Lu, Yiming Li, Chen Feng

专题命中 GUI与屏幕智能体：MLLM(abstract)；分类 cs.CV

2305.08144 2024-06-14 cs.AI 57%

Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction

Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, Kai Yu

专题命中 GUI与屏幕智能体：vision language model(abstract)；分类 cs.AI

2406.01014 2024-06-04 cs.CL cs.CV 57%

Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang

专题命中 GUI与屏幕智能体：MLLM(abstract)；分类 cs.CV

Comments 22 pages, 11 figures, 10 Tables

2402.07939 2024-05-24 cs.HC cs.AI cs.CL 57%

UFO: A UI-Focused Agent for Windows OS Interaction

Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

专题命中 GUI与屏幕智能体：grounding(abstract)；分类 cs.AI

2405.05363 2024-05-10 cs.CV cs.RO 57%

LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigation

Tianrui Guan, Yurou Yang, Harry Cheng, Muyuan Lin, Richard Kim, Rajasimman Madhivanan, Arnie Sen, Dinesh Manocha

专题命中 GUI与屏幕智能体：VLM(abstract)；分类 cs.CV

Comments Accepted to ICRA 2024

1. GUI与屏幕智能体 1586 篇

Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability

NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation

Cognitive Planning for Object Goal Navigation using Generative AI Models

EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

Reliable Semantic Understanding for Real World Zero-shot Object Goal Navigation

Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation

Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology

VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values

LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulation

Improving Zero-Shot ObjectNav with Generative Communication

CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

Towards Unified Facial Action Unit Recognition Framework by Large Language Models

RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation

Text-Enhanced Zero-Shot Action Recognition: A training-free approach

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments

OpenVLA: An Open-Source Vision-Language-Action Model

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigation

PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos

PALM: Predicting Actions through Language Models

Prioritized Semantic Learning for Zero-shot Instance Navigation

Local Action-Guided Motion Diffusion Model for Text-to-Motion Generation

MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

Tell Me Where You Are: Multimodal LLMs Meet Place Recognition

Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction

Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

UFO: A UI-Focused Agent for Windows OS Interaction

LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigation