arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04466cs.SEcs.AIcs.LGcs.OS
收藏

面向 NVIDIA Holoscan 中医疗 AI 系统的确定性端到端延迟

Towards Deterministic End-to-end Latency for Medical AI Systems in NVIDIA Holoscan

NVIDIA(英伟达)

Soham Sinha, Shekhar Dwivedi, Mahdi Azizian

AI总结:

本文针对 NVIDIA Holoscan 平台,提出一种利用 CUDA MPS 空间分区和 GPU 隔离的系统设计,以降低并发医疗 AI 应用的端到端延迟,提升确定性,并显著改善延迟指标。

摘要:

将 AI 和 ML 技术引入医疗设备已彻底改变了医疗诊断和治疗。医疗设备制造商热衷于通过将多个应用整合到单一平台上来最大化 AI 和 ML 带来的优势。然而,多个 AI 应用(每个应用都有自己的可视化组件)的并发执行会导致不可预测的端到端延迟,这主要是由于 GPU 资源争用所致。为了缓解这一问题,制造商通常为不同的 AI 应用部署独立的工作站,从而增加了财务、能源和维护成本。本文在 NVIDIA 的 Holoscan 平台(一个用于流式传输传感器数据和图像的实时 AI 系统)背景下解决了这些挑战。我们提出了一种针对异构 GPU 工作负载(涵盖计算和图形任务)优化的系统设计。我们的设计利用 CUDA MPS 对计算工作负载进行空间分区,并将计算和图形处理隔离到单独的 GPU 上。我们通过使用真实的 Holoscan 医疗设备应用进行实证评估,展示了在各种端到端延迟确定性指标上的显著性能改进。例如,与单 GPU 基线相比,所提出的设计在多达五个并发的内窥镜工具跟踪 AI 应用中将最大延迟降低了 21-30%,并将延迟分布平坦度提高了 17-25%。与默认的多 GPU 设置相比,我们的优化通过将 GPU 利用率提高 42%,在多达六个并发应用时将最大延迟降低了 35%。本文为边缘计算领域(包括医疗系统)中的 AI 应用提供了清晰的设计见解,在这些领域中,并发和异构 GPU 工作负载的性能可预测性是一个关键要求。

英文摘要:

The introduction of AI and ML technologies into medical devices has revolutionized healthcare diagnostics and treatments. Medical device manufacturers are keen to maximize the advantages afforded by AI and ML by consolidating multiple applications onto a single platform. However, concurrent execution of several AI applications, each with its own visualization components, leads to unpredictable end-to-end latency, primarily due to GPU resource contentions. To mitigate this, manufacturers typically deploy separate workstations for distinct AI applications, thereby increasing financial, energy, and maintenance costs. This paper addresses these challenges within the context of NVIDIA's Holoscan platform, a real-time AI system for streaming sensor data and images. We propose a system design optimized for heterogeneous GPU workloads, encompassing both compute and graphics tasks. Our design leverages CUDA MPS for spatial partitioning of compute workloads and isolates compute and graphics processing onto separate GPUs. We demonstrate significant performance improvements across various end-to-end latency determinism metrics through empirical evaluation with real-world Holoscan medical device applications. For instance, the proposed design reduces maximum latency by 21-30% and improves latency distribution flatness by 17-25% for up to five concurrent endoscopy tool tracking AI applications, compared to a single-GPU baseline. Against a default multi-GPU setup, our optimizations decrease maximum latency by 35% for up to six concurrent applications by improving GPU utilization by 42%. This paper provides clear design insights for AI applications in the edge-computing domain including medical systems, where performance predictability of concurrent and heterogeneous GPU workloads is a critical requirement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑