Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigrid

Characterizing Concurrency Mechanisms for NVIDIA GPUs under Deep Learning Workloads

We investigate the performance of the concurrency mechanisms available on NVIDIA's new Ampere GPU microarchitecture under deep learning training and inference workloads. In contrast to previous studies that treat the GPU as a black box, we examine scheduling at the microarchitectural level. We find that the lack of fine-grained preemption mechanisms, robust task prioritization options, and contention-aware thread block placement policies limits the effectiveness of NVIDIA's concurrency mechanisms. In summary, the sequential nature of deep learning workloads and their fluctuating resource requi…

面向LLM教师蒸馏标注的可扩展流水线：工作窃取作业调度与内存感知GPU并发控制

A Scalable Pipeline for LLM-Teacher Distillation Labeling: Work-Stealing Job Scheduling and Memory-Aware GPU Concurrency

该研究提出了一种用于LLM教师蒸馏标注的可扩展流水线，通过工作窃取作业调度、内存感知GPU并发控制及重标注基准方法，解决了标注的质量成本与GPU集群负载问题，性能优于静态分片且可复现。

Distributed Multigrid Neural Solvers on Megavoxel Domains

We consider the distributed training of large-scale neural networks that serve as PDE solvers producing full field outputs. We specifically consider neural solvers for the generalized 3D Poisson equation over megavoxel domains. A scalable framework is presented that integrates two distinct advances. First, we accelerate training a large model via a method analogous to the multigrid technique used in numerical linear algebra. Here, the network is trained using a hierarchy of increasing resolution inputs in sequence, analogous to the 'V', 'W', 'F', and 'Half-V' cycles used in multigrid approache…

MOSAIC: 通过自适应聚合和推理并发的高效混合智能体调度

MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrency

针对混合智能体系统在有限GPU资源下的负载不均衡问题，提出基于整数线性规划调度器和置信度感知自适应聚合的MOSAIC框架，实现最高2.5倍专家阶段、4.23倍聚合阶段和1.7~2.3倍端到端加速，精度损失在0.1个百分点内。

用多网格方法并行化人工神经网络训练运行

Parallelizing Over Artificial Neural Network Training Runs with Multigrid

本文提出一种多网格降时（MGRIT）算法，用于并行化神经网络训练运行，以解决训练阶段的瓶颈问题，通过将训练视为演化方程来实现并行计算。

CoPRIS：通过并发控制的部分展开与重要性采样实现高效稳定的强化学习

CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Sampling

CoPRIS通过并发控制的部分展开、提前终止与轨迹重用，结合跨阶段重要性采样校正，解决了LLM强化学习中长尾轨迹导致的GPU空闲问题，在数学推理基准上实现最高1.94倍训练加速且性能不降。