Gensors:利用多模态基础模型与推理创作个性化视觉传感器
Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning
AI总结:
针对用户难以表达需求和调试基于多模态大语言模型的个性化AI传感器的问题,本文提出Gensors系统,通过基于标准的推理、并行测试与用例建议,显著提升了用户在定义和调试传感器时的控制感、理解度与沟通便捷性。
摘要:
多模态大语言模型(MLLMs)凭借其广博的世界知识和推理能力,为终端用户创造了能够对复杂情况进行推理的个性化AI传感器的独特机会。用户可以用自然语言描述所需的传感任务(例如,“如果我的幼儿在捣乱就发出警报”),由MLLM分析摄像头画面并在几秒内做出响应。在一项形成性研究中,我们发现用户看到了定义自己传感器的巨大价值,但很难表达他们独特的个人需求,并且仅通过提示进行传感器调试也面临困难。为了应对这些挑战,我们开发了Gensors,这是一个赋能用户利用MLLMs的推理能力来定义定制传感器的系统。Gensors 1)通过自动生成和手动创建的传感器标准协助用户启发需求,2)允许用户并行隔离和测试单个标准以促进调试,3)根据用户提供的图像建议额外标准,4)提出测试用例以帮助用户对潜在不可预见的场景进行传感器的“压力测试”。在一项用户研究中,参与者报告称,使用Gensors定义传感器时,他们感受到了显著增强的控制感、理解度和沟通便捷性。除了克服模型局限性外,Gensors还支持用户通过基于标准的推理来调试、启发需求并向传感器表达独特的个人需求;它还通过暴露被忽视的标准和揭示意想不到的失败模式,帮助发现用户的“盲点”。最后,我们讨论了MLLMs的独特特征(如幻觉和不一致的响应)如何影响传感器创建过程。这些发现为未来直观且可由日常用户定制的智能传感系统的设计做出了贡献。
英文摘要:
Multimodal large language models (MLLMs), with their expansive world knowledge and reasoning capabilities, present a unique opportunity for end-users to create personalized AI sensors capable of reasoning about complex situations. A user could describe a desired sensing task in natural language (e.g., "alert if my toddler is getting into mischief"), with the MLLM analyzing the camera feed and responding within seconds. In a formative study, we found that users saw substantial value in defining their own sensors, yet struggled to articulate their unique personal requirements and debug the sensors through prompting alone. To address these challenges, we developed Gensors, a system that empowers users to define customized sensors supported by the reasoning capabilities of MLLMs. Gensors 1) assists users in eliciting requirements through both automatically-generated and manually created sensor criteria, 2) facilitates debugging by allowing users to isolate and test individual criteria in parallel, 3) suggests additional criteria based on user-provided images, and 4) proposes test cases to help users "stress test" sensors on potentially unforeseen scenarios. In a user study, participants reported significantly greater sense of control, understanding, and ease of communication when defining sensors using Gensors. Beyond addressing model limitations, Gensors supported users in debugging, eliciting requirements, and expressing unique personal requirements to the sensor through criteria-based reasoning; it also helped uncover users' "blind spots" by exposing overlooked criteria and revealing unanticipated failure modes. Finally, we discuss how unique characteristics of MLLMs--such as hallucinations and inconsistent responses--can impact the sensor-creation process. These findings contribute to the design of future intelligent sensing systems that are intuitive and customizable by everyday users.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。