arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.07034cs.ROcs.AIcs.CV
收藏

KITE:基于关键帧的令牌化证据用于基于视觉语言模型的机器人故障分析

KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis

Australian Institute for Machine Learning (AIML), Adelaide University(澳大利亚机器学习研究所(AIML),阿德莱德大学)

Mehdi Hosseinzadeh, King Hang Wong, Feras Dayoub

AI总结:

KITE通过将长机器人执行视频转化为紧凑可解释的令牌化证据,提升基于VLM的机器人故障检测与分析能力,尤其在模拟故障检测中表现优异。

摘要:

我们提出了KITE,一种无需训练的关键帧锚定、布局导向的前端,将长机器人执行视频转化为紧凑、可解释的令牌化证据,供视觉语言模型(VLM)使用。KITE将每个轨迹压缩为少量运动显著的关键帧,配以包含相对物体布局、轴、时间戳和检测置信度的鸟瞰图表示。这些视觉线索与机器人配置和场景上下文令牌序列化为统一提示,使同一前端可支持故障检测、识别、定位、解释和纠正。在RoboFAC基准测试中,KITE与Qwen2.5-VL相比在无训练设置下表现更优,尤其在模拟故障检测、识别和定位上提升显著,同时与RoboFAC调优基线竞争。小规模QLoRA微调进一步提升解释和纠正质量。我们还报告了在真实双臂机器人上的定性结果,证明KITE作为结构化且可解释的机器人故障分析前端的实用性。代码和模型已发布在项目页面:https://m80hz.github.io/kite/

英文摘要:

We present KITE, a training-free, keyframe-anchored, layout-grounded front-end that converts long robot-execution videos into compact, interpretable tokenized evidence for vision-language models (VLMs). KITE distills each trajectory into a small set of motion-salient keyframes with open-vocabulary detections and pairs each keyframe with a schematic bird's-eye-view (BEV) representation that encodes relative object layout, axes, timestamps, and detection confidence. These visual cues are serialized with robot-profile and scene-context tokens into a unified prompt, allowing the same front-end to support failure detection, identification, localization, explanation, and correction with an off-the-shelf VLM. On the RoboFAC benchmark, KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5-VL in the training-free setting, with especially large gains on simulation failure detection, identification, and localization, while remaining competitive with a RoboFAC-tuned baseline. A small QLoRA fine-tune further improves explanation and correction quality. We also report qualitative results on real dual-arm robots, demonstrating the practical applicability of KITE as a structured and interpretable front-end for robot failure analysis. Code and models are released on our project page: https://m80hz.github.io/kite/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑