Large language models (LLMs) have demonstrated exceptional abilities across various domains. However, utilizing LLMs for ubiquitous sensing applications remains challenging as existing text-prompt methods show significant performance degradation when handling long sensor data sequences. We propose a visual prompting approach for sensor data using multimodal LLMs (MLLMs). We design a visual prompt that directs MLLMs to utilize visualized sensor data alongside the target sensory task descriptions. Additionally, we introduce a visualization generator that automates the creation of optimal visualizations tailored to a given sensory task, eliminating the need for prior task-specific knowledge. We evaluated our approach on nine sensory tasks involving four sensing modalities, achieving an average of 10% higher accuracy than text-based prompts and reducing token costs by 15.8 times. Our findings highlight the effectiveness and cost-efficiency of visual prompts with MLLMs for various sensory tasks. The source code is available at https://github.com/diamond264/ByMyEyes.
Better Eyes, Better Thoughts: Why Vision Chain-of-Thought Fails in Medicine
Department of Computer Science, City University of Hong Kong (Dongguan)（香港城市大学（东莞）计算机科学系）；College of Computing and Data Science (CCDS), Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院）
Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
Beijing Jiaotong University（北京交通大学）；Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA)（基础模型研究中心、自动化研究所、中国科学院（CASIA））；Tencent Robotics X（腾讯机器人X）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST)（计算机科学与工程系、香港科学与技术大学（HKUST））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
Smart Eyes for Silent Threats: VLMs and In-Context Learning for THz Imaging
Data and Web Science Group, University of Mannheim（曼海姆大学数据与网络科学组）；Max-Planck-Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克信息研究所，萨尔兰州信息校区）
Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs
Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Georgia Institute of Technology（佐治亚理工学院）；Nanjing University of Science and Technology（南京理工大学）；University of Oxford（牛津大学）；The Ohio State University（俄亥俄州立大学）
Grounding Large Language Models in Reaction Knowledge Graphs for Synthesis Retrieval
Department of Intelligent Systems, Delft University of Technology（智能系统系，代尔夫特理工大学）；Department of Chemical Engineering, Delft University of Technology（化学工程系，代尔夫特理工大学）
Visual Prompting in Multimodal Large Language Models: A Survey
UC San Diego（加州大学圣地亚哥分校）；Dartmouth College（达特茅斯学院）；Rutgers University（罗格斯大学）；UC Los Angeles（加州大学洛杉矶分校）；Adobe Research（奥多比研究院）；The University of New South Wales（新南威尔士大学）；CSIRO’s Data61（联邦科学与工业研究组织Data61）
Generalist Scanner Meets Specialist Locator: A Synergistic Coarse-to-Fine Framework for Robust GUI Grounding
University of California, San Diego（加州大学圣地亚哥分校）；ByteDance（字节跳动）；University of California, Merced（加州大学默塞德分校）；University of Southern California（南加州大学）；University at Buffalo（布法罗大学）；The University of Queensland（昆士兰大学）
University of Freiburg（弗赖堡大学）；University of Technology Nuremberg（纽伦堡工业大学）
Recent works have shown that Large Language Models (LLMs) can be applied to ground natural language to a wide variety of robot skills. However, in practice, learning multi-task, language-conditioned robotic skills typically requires large-scale data collection and frequent human intervention to reset the environment or help correcting the current policies. In this work, we propose a novel approach to efficiently learn general-purpose language-conditioned robot skills from unstructured, offline and reset-free data in the real world by exploiting a self-supervised visuo-lingual affordance model,…
3DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o
Shanghai AI Lab（上海人工智能实验室）；Dalian University of Technology（大连理工大学）；School of Geodesy and Geomatics, Wuhan University（武汉大学测绘学院）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
提出 3DAxisPrompt，通过 3D 坐标轴与 SAM 掩码为 GPT-4o 提供几何先验，并在四个数据集上验证其 3D 定位与推理能力。
Seeing Through Experts Eyes A Foundational Vision Language Model Trained on Radiologists Gaze and Reasoning
Bioengineering Department and Imperial-X（生物工程系和Imperial-X）；Imperial College London（帝国理工学院伦敦分校）；College of physics and information engineering, Fuzhou University（福州大学物理与信息工程学院）；Department of Surgery and Cancer, Imperial College London（外科与癌症部门，帝国理工学院伦敦分校）；National Heart and Lung Institute, Imperial College London（国家心脏和肺研究所，帝国理工学院伦敦分校）；Cardiovascular Research Centre, Royal Brompton Hospital（心血管研究中心，皇家布里顿医院）；School of Biomedical Engineering & Imaging Sciences, King’s College London（生物医学工程与成像科学学院，国王学院伦敦分校）
VisDoT : Enhancing Visual Reasoning through Human-Like Interpretation Grounding and Decomposition of Thought
Department of Computer Science and Artificial Intelligence, Dongguk University（东国大学计算机科学与人工智能系）；Department of Electronics and Electrical Engineering, Dongguk University（东国大学电子与电气工程系）
Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Helmholtz AI（亥姆霍兹人工智能研究所）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Imperial College London（伦敦帝国学院）；King’s College London（伦敦国王学院）
PHAnToM: Persona-based Prompting Has An Effect on Theory-of-Mind Reasoning in Large Language Models
Institute of Data Science（数据科学研究所）；NUS Centre for Trusted Internet and Community, National University of Singapore（新加坡国立大学可信互联网与社区中心）；Amazon（亚马逊）；Stanford University（斯坦福大学）；Tsinghua University（清华大学）
Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language
Rakuten Institute of Technology（乐天技术研究所）；Rakuten Group, Inc（乐天集团）；National Institute of Advanced Industrial Science and Technology（日本产业技术综合研究所）；Graduate School of Engineering, The University of Tokyo（东京大学大学院工学系研究科）
Robotics at Google（谷歌机器人部门）；Everyday Robots（每日机器人公司）
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to…
Topology-Aware Reasoning over Incomplete Knowledge Graph with Graph-Based Soft Prompting
Chalmers University of Technology and University of Gothenburg, Sweden（楚姆勒大学技术学院和哥德堡大学，瑞典）；Technical University of Denmark, Kgs. Lyngby, Denmark（丹麦技术大学，基斯勒吕辛，丹麦）
Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding
Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Macquarie University（麦考瑞大学）；Hanoi University of Science and Technology（河内科学技术大学）；University of Wollongong（沃林根大学）；Flinders University（弗林德斯大学）
WildRoadBench: A Wild Aerial Road-Damage Grounding Benchmark for Vision-Language Models and Autonomous Agents
University of Electronic Science and Technology of China（电子科技大学）；National University of Singapore（新加坡国立大学）；De Artificial Intelligence Lab（德人工智能实验室）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；University of Science and Technology of China（中国科学技术大学）
Enhancing Vision-Language Models for Autonomous Driving through Task-Specific Prompting and Spatial Reasoning
University of Chinese Academy of Sciences（中国科学院大学）；Technology and Engineering Center for Space Utilization, Chinese Academy of Sciences（中国科学院空间利用技术与工程中心）
ChartPoint: Guiding MLLMs with Grounding Reflection for Chart Reasoning
Tsinghua University（清华大学）；International Digital Economy Academy（国际数字经济学院）；Beihang University（北航）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
