Kafka-ML:将数据流与ML/AI框架连接
Kafka-ML: connecting the data stream with ML/AI frameworks
摘要:
机器学习(ML)和人工智能(AI)依赖于数据源来通过其算法进行训练、改进和预测。随着数字革命和物联网等当前范式的发展,这些信息正在从静态数据转变为连续的数据流。然而,现在使用的大多数ML/AI框架都没有为这场革命做好充分的准备。在本文中,我们提出了Kafka-ML,这是一个开源框架,可以通过数据流(Apache Kafka)管理TensorFlow ML/AI管道。Kafka-ML提供了一个可访问且用户友好的Web用户界面,用户可以轻松定义ML模型,然后训练,评估和部署它们进行推理。Kafka-ML本身及其部署的组件通过容器化技术进行全面管理,这确保了其可移植性和易于分发以及容错和高可用性等其他功能。最后,介绍了一种新的方法来管理和重用数据流,这可能导致(不)利用数据存储和文件系统。
英文摘要:
Machine Learning (ML) and Artificial Intelligence (AI) have a dependency on data sources to train, improve and make predictions through their algorithms. With the digital revolution and current paradigms like the Internet of Things, this information is turning from static data into continuous data streams. However, most of the ML/AI frameworks used nowadays are not fully prepared for this revolution. In this paper, we proposed Kafka-ML, an open-source framework that enables the management of TensorFlow ML/AI pipelines through data streams (Apache Kafka). Kafka-ML provides an accessible and user-friendly Web User Interface where users can easily define ML models, to then train, evaluate and deploy them for inference. Kafka-ML itself and its deployed components are fully managed through containerization technologies, which ensure its portability and easy distribution and other features such as fault-tolerance and high availability. Finally, a novel approach has been introduced to manage and reuse data streams, which may lead to the (no) utilization of data storage and file systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。