The advantages of pre-trained large language models (LLMs) are apparent in a variety of language processing tasks. But can a language model's knowledge be further harnessed to effectively disambiguate objects and navigate decision-making challenges within the realm of robotics? Our study reveals the LLM's aptitude for solving complex decision making challenges that are often previously modeled by Partially Observable Markov Decision Processes (POMDPs). A pivotal focus of our research is the object disambiguation capability of LLMs. We detail the integration of an LLM into a tabletop environment disambiguation task, a decision making problem where the robot's task is to discern and retrieve a user's desired object from an arbitrarily large and complex cluster of objects. Despite multiple query attempts with zero-shot prompt engineering (details can be found in the Appendix), the LLM struggled to inquire about features not explicitly provided in the scene description. In response, we have developed a few-shot prompt engineering system to improve the LLM's ability to pose disambiguating queries. The result is a model capable of both using given features when they are available and inferring new relevant features when necessary, to successfully generate and navigate down a precise decision tree to the correct object--even when faced with identical options.
Enhancing Speech Instruction Understanding and Disambiguation in Robotics via Speech Prosody
IT University of Copenhagen(丹麦技术大学);University of Florida(佛罗里达大学);Brown University(布朗大学);Intelligent Agents Research Group(智能代理研究组);Intelligent Robot Lab(智能机器人实验室)
Vision-Based Reasoning with Topology-Encoded Graphs for Anatomical Path Disambiguation in Robot-Assisted Endovascular Navigation
Department of Control Science and Engineering, College of Electronics and Information Engineering, and Shanghai Institute of Intelligent Science and Technology, Tongji University(控制科学与工程系,电子与信息工程学院和智能科学与技术上海研究院,同济大学);Department of Electronic Engineering, Faculty of Engineering, The Chinese University of Hong Kong(电子工程系,工程学院,香港中文大学);Shanghai Operation Robot Co., Ltd.(上海操作机器人有限公司);Department of Cardiology, Zhongshan Hospital, Fudan University, Shanghai Institute of Cardiovascular Diseases, National Clinical Research Center for Interventional Medicine(心内科,复旦大学中山医院,上海心血管疾病研究所,介入医学临床研究中心);State Key Laboratory of Cardiovascular Diseases and Medical Innovation Center, Shanghai East Hospital, School of Medicine, Tongji University(心血管疾病国家重点实验室和医学创新中心,上海东医院,同济大学医学院)
Multi-Functional Embedding Models for Funder Name Disambiguation in Scientific Publication Records
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Michigan(密歇根大学);Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所)
REFLEX: Metacognitive Reasoning for Reflective Zero-Shot Robotic Planning with Large Language Models
School of Applied and Creative Computing, Purdue University(应用与创意计算学院,普渡大学);Bowen School of Construction, Purdue University(建设学院,普渡大学);School of Engineering Technology, Purdue University(工程技术学院,普渡大学);Department of Technology Leadership and Innovation, Purdue University(技术领导与创新部门,普渡大学)
Assessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya, Aichi, Japan(Nagoya大学微纳米机械科学与工程系);Institutes of Innovation for Future Society, Nagoya University, Nagoya, Aichi, Japan(Nagoya大学未来社会创新研究所)
Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue
University of Michigan(密歇根大学);University of Arkansas(阿肯色大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of Copenhagen(哥本哈根大学);IT University of Copenhagen(哥本哈根信息技术大学)
Small Language Model Helps Resolve Semantic Ambiguity of LLM Prompt
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);Department of Electronic Engineering, Kyung Hee University(庆熙大学电子工程系)