视觉大模型 / VLM

Unifying Global and Local Scene Entities Modelling for Precise Action Spotting

Kim Hoang Tran, Phuc Vuong Do, Ngoc Quoc Ly, Ngan Le

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted to IJCNN 2024

Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation

Vishnu Sashank Dorbala, James F. Mullen, Dinesh Manocha

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments 10 pages

Journal ref IEEE Robotics and Automation Letters 9.5 (2024) 4083-4090

Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level

Chenxu Wang, Bin Dai, Huaping Liu, Baoyuan Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition

Tongjia Chen, Hongshan Yu, Zhengeng Yang, Zechuan Li, Wei Sun, Chen Chen

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Technical report. Project Page: https://tomchen-ctj.github.io/OST/

MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception

Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, Jing Shao

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Comments Accepted to CVPR2024

Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation

Razvan-George Pasca, Alexey Gavryushin, Muhammad Hamza, Yen-Ling Kuo, Kaichun Mo, Luc Van Gool, Otmar Hilliges, Xi Wang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skill

Sangwoo Shin, Daehee Lee, Minjong Yoo, Woo Kyung Kim, Honguk Woo

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

Comments ICML-2023 Camera Ready Version

Target-Grounded Graph-Aware Transformer for Aerial Vision-and-Dialog Navigation

Yifei Su, Dong An, Yuan Xu, Kehan Chen, Yan Huang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 1st Place Solution for the AVDN Challenge in ICCV CLVL 2023

VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation

Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, Bernadette Bucher

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition

Yunjiao Zhou, Jianfei Yang, Han Zou, Lihua Xie

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Preprint manuscript in submission

PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers' Workflows

Savvas Petridis, Michael Terry, Carrie J. Cai

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Videoprompter: an ensemble of foundational models for zero-shot video understanding

Adeel Yousaf, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Mubarak Shah

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Video Action Recognition with Attentive Semantic Units

Yifei Chen, Dapeng Chen, Ruijin Liu, Hao Li, Wei Peng

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV

Comments Accepted at ICCV 2023

Journal ref Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 10170-10180

RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction

Isaac Kasahara, Shubham Agrawal, Selim Engin, Nikhil Chavan-Dafle, Shuran Song, Volkan Isler

专题命中 GUI与屏幕智能体 ：visual language model(abstract)；分类 cs.CV

Masked Diffusion with Task-awareness for Procedure Planning in Instructional Videos

Fen Fang, Yun Liu, Ali Koksal, Qianli Xu, Joo-Hwee Lim

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments 7 pages (main text excluding references), 3 figures, 7 tables

DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion

Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, Yi Yang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments 12 pages, 7 figures

Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation

Haitian Zeng, Xiaohan Wang, Wenguan Wang, Yi Yang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 10 pages, 4 figures

Referring to Screen Texts with Voice Assistants

Shruti Bhargava, Anand Dhoot, Ing-Marie Jonsson, Hoang Long Nguyen, Alkesh Patel, Hong Yu, Vincent Renkens

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments 7 pages, Accepted to ACL Industry Track 2023

InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language

Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, Jiashuo Yu, Kunchang Li, Zhe Chen, Xue Yang, Xizhou Zhu, Yali Wang, Limin Wang, Ping Luo, Jifeng Dai, Yu Qiao

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Technical Report

Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, Dhruv Mahajan

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments CVPR 2023

Lexi: Self-Supervised Learning of the UI Language

Pratyay Banerjee, Shweti Mahajan, Kushal Arora, Chitta Baral, Oriana Riva

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments EMNLP (Findings) 2022

Knowledge Prompting for Few-shot Action Recognition

Yuheng Shi, Xinxiao Wu, Hanxi Lin

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Structured Exploration Through Instruction Enhancement for Object Navigation

Matthias Hutsebaut-Buysse, Kevin Mets, Tom De Schepper, Steven Latré

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments Paper accepted to the BNAIC/BeNeLearn 2022 conference

ULN: Towards Underspecified Vision-and-Language Navigation

Weixi Feng, Tsu-Jui Fu, Yujie Lu, William Yang Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments EMNLP 2022

Ground then Navigate: Language-guided Navigation in Dynamic Scenes

Kanishk Jain, Varun Chhangani, Amogh Tiwari, K. Madhava Krishna, Vineet Gandhi

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Less is More: Generating Grounded Navigation Instructions from Landmarks

Su Wang, Ceslee Montgomery, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, Peter Anderson

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments CVPR 2022 Camera-ready

Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation

Yicong Hong, Zun Wang, Qi Wu, Stephen Gould

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation

Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, Ivan Laptev

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Learning to Stop: A Simple yet Effective Approach to Urban Vision-Language Navigation

Jiannan Xiang, Xin Eric Wang, William Yang Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments Findings of EMNLP 2020

Mapping Natural Language Instructions to Mobile UI Action Sequences

Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, Jason Baldridge

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.LG

Comments Annual Conference of the Association for Computational Linguistics (ACL 2020)