AmadeusGPT:用于交互式动物行为分析的自然语言接口
AmadeusGPT: a natural language interface for interactive animal behavioral analysis
AI总结:
AmadeusGPT通过双记忆机制增强大型语言模型,将自然语言行为描述转化为可执行代码,实现无需编程的交互式动物行为分析,并在MABE 2022挑战中达到最先进性能。
摘要:
量化和分析动物行为的过程涉及将动物行为的自然描述语言转化为机器可读代码。然而,在没有深入理解动物行为和技术性机器学习知识的情况下,编码行为分析通常具有挑战性。为了缩小这一差距,我们引入了AmadeusGPT:一个将自然语言行为描述转化为机器可执行代码的自然语言接口。诸如GPT3.5和GPT4这样的大型语言模型(LLMs)允许基于语言的交互式查询,这可能非常适合进行交互式行为分析。然而,这些LLMs的理解能力受限于上下文窗口大小,这使其无法记住较远的对话。为了克服上下文窗口的限制,我们实现了一种新颖的双记忆机制,允许短期和长期记忆之间通过符号作为上下文指针进行检索和保存的通信。具体而言,用户直接使用基于语言的行为定义,我们增强的GPT基于核心AmadeusGPT API开发代码,该API包含机器学习、计算机视觉、时空推理和可视化模块。用户随后可以交互式地细化结果,并根据需要无缝添加新的行为模块。我们对AmadeusGPT进行了基准测试,并展示了在MABE 2022行为挑战任务上能够产生最先进的性能。注意,最终用户无需编写任何代码即可实现这一点。因此,总体上,AmadeusGPT提出了一种新颖的方式,将深层生物学知识、大型语言模型和核心计算机视觉模块融合到一个更自然的智能系统中。代码和演示可在https://github.com/AdaptiveMotorControlLab/AmadeusGPT找到。
英文摘要:
The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of animal behavior and technical machine learning knowledge. To limit this gap, we introduce AmadeusGPT: a natural language interface that turns natural language descriptions of behaviors into machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4 allow for interactive language-based queries that are potentially well suited for making interactive behavior analysis. However, the comprehension capability of these LLMs is limited by the context window size, which prevents it from remembering distant conversations. To overcome the context window limitation, we implement a novel dual-memory mechanism to allow communication between short-term and long-term memory using symbols as context pointers for retrieval and saving. Concretely, users directly use language-based definitions of behavior and our augmented GPT develops code based on the core AmadeusGPT API, which contains machine learning, computer vision, spatio-temporal reasoning, and visualization modules. Users then can interactively refine results, and seamlessly add new behavioral modules as needed. We benchmark AmadeusGPT and show we can produce state-of-the-art performance on the MABE 2022 behavior challenge tasks. Note, an end-user would not need to write any code to achieve this. Thus, collectively AmadeusGPT presents a novel way to merge deep biological knowledge, large-language models, and core computer vision modules into a more naturally intelligent system. Code and demos can be found at: https://github.com/AdaptiveMotorControlLab/AmadeusGPT.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。