Gyrokinetic codes in plasma physics need outstanding computational resources to solve increasingly complex problems, requiring the effective exploitation of cutting-edge HPC architectures. This paper focuses on the enabling of ORB5, a state-of-the-art, first-principles-based gyrokinetic code, on modern parallel hybrid multi-core, multi-GPU systems. ORB5 is a Lagrangian, Particle-In-Cell (PIC), finite element, global, electromagnetic code, originally implementing distributed parallelism through MPI-based on domain decomposition and domain cloning. In order to support multi/many cores devices, the code has been completely refactored. Data structures have been re-designed to ensure efficient memory access, enhancing data locality. Multi-threading has been introduced through OpenMP on the CPU and adopting OpenACC to support GPU acceleration. MPI can still be used in combination with the two approaches. The performance results obtained using the full production ORB5 code on the Summit system at ORNL, on Piz Daint at CSCS and on the Marconi system at CINECA are presented, showing the effectiveness and performance portability of the adopted solutions: the same source code version was used to produce all results on all architectures.
We present an energy-conserving discontinuous Galerkin scheme for the full-$f$ electromagnetic gyrokinetic system in the long-wavelength limit. We use the symplectic formulation and solve directly for $\partial A_\parallel/\partial t$, the inductive component of the parallel electric field, using a generalized Ohm's law derived directly from the gyrokinetic equation. Linear benchmarks are performed to verify the implementation and show that the scheme avoids the Ampère cancellation problem. We perform a nonlinear electromagnetic simulation in a helical open-field-line system as a rough model o…
While in recent years, gyrokinetic simulations have become the workhorse for theoretical turbulence and transport studies in the plasma core, their application to the edge and scrape-off layer (SOL) region presents significant challenges. In particular, steep density and temperature gradients as well as large fluctuation amplitudes call for a "full-f" treatment. To specifically study problems in the SOL region, the gyrokinetic particle-in-cell (PIC) code PICLS has been developed. The code is based on an electrostatic full-f model with linearised field equations and uses kinetic electrons. Here…
Applying gyrokinetic simulations for theoretical turbulence and transport studies to the plasma edge and scrape-off layer (SOL) presents significant challenges. To in particular account for steep density and temperature gradients in the SOL, the "full-f" code PICLS was developed. PICLS is a gyrokinetic particle-in-cell (PIC) code and is based on an electrostatic model with a linearized field equation and uses kinetic electrons. In previously published results we were applying PICLS to the well-studied 1D parallel transport problem during an edge-localized mode (ELM) in the SOL without collisio…
Turbulence is shown to be critical to the onset and evolution of the neoclassical tearing mode, affecting both its growth and rotation. The interaction is here studied for the first time in the three dimensional, toroidal gyrokinetic framework. Turbulent fluctuations do not destroy the growing island early in its development, which maintains a coherent form as it grows, in fact the island is seeded and its rotation frequency determined, by nonlinear interaction. This process provides an initial structure that is of the order of an ion gyro-radius wide, allowing the island to rapidly reach a la…
TRIMEG-GKX: an electromagnetic gyrokinetic particle code with a Piecewise Field-Aligned Finite Element Method for Micro- and Macro-Instability Studies in Tokamak Core Plasmas
Experimental data on electromagnetic fluctuations in DIII-D, made available by the Faraday-effect Radial Interferometer-Polarimeter (RIP) diagnostic, is examined in comparison with detailed gyrokinetic simulations using Gyrokinetic Electromagnetic Numerical Experiment (GENE). The diagnostic has the unique capability of making internal measurements of fluctuating magnetic fields $\frac{\int n_e δB_r dR}{\int n_e dR}$. Local linear simulations identify microtearing modes (MTMs) over a substantial range of toroidal mode numbers (peaking at $n=15$) with frequencies in good agreement with the exper…
The gyrokinetic simulation code AstroGK is developed to study fundamental aspects of kinetic plasmas and for applications mainly to astrophysical problems. AstroGK is an Eulerian slab code that solves the electromagnetic Gyrokinetic-Maxwell equations in five-dimensional phase space, and is derived from the existing gyrokinetics code GS2 by removing magnetic geometry effects. Algorithms used in the code are described. The code is benchmarked using linear and nonlinear problems. Serial and parallel performance scalings are also presented.
The Gyrokinetic Toroidal Code at Princeton (GTC-P) is a highly scalable and portable particle-in-cell (PIC) code. It solves the 5D Vlasov-Poisson equation featuring efficient utilization of modern parallel computer architectures at the petascale and beyond. Motivated by the goal of developing a modern code capable of dealing with the physics challenge of increasing problem size with sufficient resolution, new thread-level optimizations have been introduced as well as a key additional domain decomposition. GTC-P's multiple levels of parallelism, including inter-node 2D domain decomposition and …
With the advent of the gyrokinetic formalism, recent developments in low-noise nonlinear $δf$ methods, and enormous gains in computing power, large-scale gyrokinetic simulations have become an important tool for improved understanding of anomalous transport in tokamaks. Simulating the non-linear behaviour requires solving for the perturbations of distrbution function in five dimensions. We use a particle-in-cell approach to solve the equations via the non-linear characteristic method. The code has been parallelized for a variety of architecures (C90, CM-5, T3D) using a 1-D domain decomposition…
For decades, the Vlasov-Darwin model has been recognized to be attractive for particle-in-cell (PIC) kinetic plasma simulations in non-radiative electromagnetic regimes, to avoid radiative noise issues and gain computational efficiency. However, the Darwin model results in an elliptic set of field equations that renders conventional explicit time integration unconditionally unstable. Here, we explore a fully implicit PIC algorithm for the Vlasov-Darwin model in multiple dimensions, which overcomes many difficulties of traditional semi-implicit Darwin PIC algorithms. The finite-difference schem…
The pullback scheme is implemented in the global gyrokinetic particle-in-cell code ORB5 [S. Jolliet et al, Comp. Phys. Comm., 177, 409 (2007)] to mitigate the cancellation problem in electromagnetic simulations. The equations and the discretisation used by the code are described. Numerical simulations of the Toroidal Alfven Eigenmodes are performed in linear and nonlinear regimes to verify the scheme. A considerable improvement in the code efficiency is observed. For the internal kink mode, it is shown that the pullback mitigation efficiently cures a numerical instability which would make the …
This paper contributes new insights into discretizing Coulomb collisions in kinetic plasma models. Building on the previous works [Carrillo et al. J. Comp. Phys. X 7:100066 (2020), Hirvijoki and Burby Phys. Plasmas 27(8):082307 (2020)], I propose deterministic discrete-time energy- and positivity-preserving, entropy-dissipating marker-particle schemes for the standard Landau collision operator and the electrostatic gyrokinetic Landau operator. In case of the standard Landau operator, the scheme preserves also the discrete-time kinetic momentum. The improvements, the extensions of the structure…
This paper investigates subcycling of particle orbits in variational, geometric particle-in-cell methods addressing the Vlasov--Maxwell system in magnetized plasmas. The purpose of subcycling is to allow different time steps for different particle species and, ideally, time steps longer than the electron gyroperiod for the global field solves while sampling the local cyclotron orbits accurately. The considered algorithms retain the electromagnetic gauge invariance of the discrete action, guaranteeing a local charge conservation law, while the variational approach provides a bounded long-time e…
An algorithm is presented that incorporates the tensor form of Maxwell's equations in a general relativistic electromagnetic particle-in-cell code. The code simplifies to Schwartzschild space-time for a non-spinning central mass. The particle advance routine uses a fourth-order Runge-Kutta algorithm to integrate the four-velocity form of Lorentz force. The current density is calculated using the curved space-time of the metric.
Comparing time and frequency domain numerical methods with Born-Rytov approximations for far-field electromagnetic scattering from single biological cells
