告知与展示：结合多模态向机器人传达操作任务
Tell and show: Combining multiple modalities to communicate manipulation tasks to a robot
AI总结：
本文提出一种融合手势和语言等多模态不确定信息并结合情境感知的机器人任务传达方法，并通过自适应熵阈值判断是否向人类询问澄清，在模拟和真实实验中验证了其鲁棒性和有效性。
摘要：
随着人机协作日益普及，需要一种更自然的与机器人沟通的方式。这包括将来自多种模态的数据与情境背景和背景知识相结合。当前的通信方法通常仅依赖单一模态，或者往往非常僵化，对缺失、错位或噪声数据不够鲁棒。本文提出了一种新方法，该方法借鉴传感器融合的思路，融合来自多种模态的不确定信息，并利用情境感知（例如考虑物体属性或场景布置）加以增强。我们首先在模拟的双模态数据集（手势和语言）上评估所提出的方案，并通过多个消融实验展示系统各组件的重要性及其对噪声、缺失或错位观测的鲁棒性。随后我们在真实装置上实现并评估该模型。在人机交互中，我们还必须考虑所选动作是否具有足够高的执行概率，或者是否应当向人类询问以进行澄清。为此，我们通过基于自适应熵的阈值方法来增强模型，该方法能够针对不同类型的交互检测合适的阈值，表现出与经过精细调优的固定阈值相似的性能。
英文摘要：
As human-robot collaboration is becoming more widespread, there is a need for a more natural way of communicating with the robot. This includes combining data from several modalities together with the context of the situation and background knowledge. Current approaches to communication typically rely only on a single modality or are often very rigid and not robust to missing, misaligned, or noisy data. In this paper, we propose a novel method that takes inspiration from sensor fusion approaches to combine uncertain information from multiple modalities and enhance it with situational awareness (e.g., considering object properties or the scene setup). We first evaluate the proposed solution on simulated bimodal datasets (gestures and language) and show by several ablation experiments the importance of various components of the system and its robustness to noisy, missing, or misaligned observations. Then we implement and evaluate the model on the real setup. In human-robot interaction, we must also consider whether the selected action is probable enough to be executed or if we should better query humans for clarification. For these purposes, we enhance our model with adaptive entropy-based thresholding that detects the appropriate thresholds for different types of interaction showing similar performance as fine-tuned fixed thresholds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
