Modern consumer electronic devices often provide intelligence services with deep neural networks. We have started migrating the computing locations of intelligence services from cloud servers (traditional AI systems) to the corresponding devices (on-device AI systems). On-device AI systems generally have the advantages of preserving privacy, removing network latency, and saving cloud costs. With the emergent of on-device AI systems having relatively low computing power, the inconsistent and varying hardware resources and capabilities pose difficulties. Authors' affiliation has started applying a stream pipeline framework, NNStreamer, for on-device AI systems, saving developmental costs and hardware resources and improving performance. We want to expand the types of devices and applications with on-device AI services products of both the affiliation and second/third parties. We also want to make each AI service atomic, re-deployable, and shared among connected devices of arbitrary vendors; we now have yet another requirement introduced as it always has been. The new requirement of "among-device AI" includes connectivity between AI pipelines so that they may share computing resources and hardware capabilities across a wide range of devices regardless of vendors and manufacturers. We propose extensions of the stream pipeline framework, NNStreamer, for on-device AI so that NNStreamer may provide among-device AI capability. This work is a Linux Foundation (LF AI and Data) open source project accepting contributions from the general public.
NNStreamer: Efficient and Agile Development of On-Device AI Systems
Samsung Research, Samsung Electronics(三星电子三星研究院)
我们提出了NNStreamer,一个软件系统,处理神经网络作为流管道的过滤器,应用流处理范式的深度神经网络应用。随着深度神经网络应用的广泛普及,一个新趋势是设备上人工智能。它是在移动设备或边缘/物联网设备上处理神经网络,而不是云服务器。新出现的隐私问题、数据传输成本和运营成本意味着对设备上AI的需求,特别是如果我们部署了大量设备。NNStreamer可有效处理设备上具有复杂数据流管道的神经网络,以最小的努力显著提高整体性能。此外,NNStreamer简化了实现,并允许直接重用现成的媒体过滤器,这大大降低了开发成本。我们已经在广泛的产品和平台上部署了NNStreamer,包括Galaxy系列和各种消费电子设备。实验结果表明,在开发成本和提高性能的管道架构和NNStreamer的减少。它是由Linux Foundation AI孵化的开源项目,可向公众提供,适用于各种硬件和软件平台。
StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation
UC Berkeley(伯克利大学);University of Tsukuba(筑波大学);International Christian University(国际基督教大学);Toyo University(东洋大学);Tokyo Institute of Technology(东京技术大学);Tohoku University(东北大学);MIT(麻省理工学院)
Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Data Science and Analytics Thrust, HKUST (GZ)(香港科技大学(广州)数据科学与分析学域);The Chinese University of Hong Kong(香港中文大学)