OpenOmni：一个用于构建面向未来的多模态对话代理的协作式开源工具
OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agents
AI总结：
为解决多模态对话代理开发与基准测试缺乏端到端解决方案的问题，本文提出了开源工具 OpenOmni，集成语音转文本、情感检测、检索增强生成和大型语言模型等技术，支持本地与云部署，便于研究人员定制流水线并平衡延迟、准确性与隐私，推动人机交互应用发展。
摘要：
多模态对话代理非常令人向往，因为它们提供自然且类人的交互。然而，目前缺乏支持协作开发和基准测试的全面端到端解决方案。尽管像 GPT-4o 和 Gemini 这样的专有系统展示了音频、视频和文本的出色集成，响应时间为 200-250ms，但在平衡延迟、准确性、成本和数据隐私方面仍存在挑战。为了更好地理解和量化这些问题，我们开发了 OpenOmni，这是一个开源的端到端流水线基准测试工具，集成了语音转文本、情感检测、检索增强生成、大型语言模型等先进技术，以及集成定制模型的能力。OpenOmni 支持本地和云部署，确保数据隐私并支持延迟和准确性基准测试。这个灵活的框架允许研究人员定制流水线，聚焦于真实瓶颈并促进快速概念验证开发。OpenOmni 可以显著增强诸如视障人士室内辅助等应用，推动人机交互的发展。我们的演示视频可在 https://www.youtube.com/watch?v=zaSiT3clWqY 查看，演示可通过 https://openomni.ai4wa.com 访问，代码可通过 https://github.com/AI4WA/OpenOmniFramework 获取。
英文摘要：
Multimodal conversational agents are highly desirable because they offer natural and human-like interaction. However, there is a lack of comprehensive end-to-end solutions to support collaborative development and benchmarking. While proprietary systems like GPT-4o and Gemini demonstrating impressive integration of audio, video, and text with response times of 200-250ms, challenges remain in balancing latency, accuracy, cost, and data privacy. To better understand and quantify these issues, we developed OpenOmni, an open-source, end-to-end pipeline benchmarking tool that integrates advanced technologies such as Speech-to-Text, Emotion Detection, Retrieval Augmented Generation, Large Language Models, along with the ability to integrate customized models. OpenOmni supports local and cloud deployment, ensuring data privacy and supporting latency and accuracy benchmarking. This flexible framework allows researchers to customize the pipeline, focusing on real bottlenecks and facilitating rapid proof-of-concept development. OpenOmni can significantly enhance applications like indoor assistance for visually impaired individuals, advancing human-computer interaction. Our demonstration video is available https://www.youtube.com/watch?v=zaSiT3clWqY, demo is available via https://openomni.ai4wa.com, code is available via https://github.com/AI4WA/OpenOmniFramework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
