AMGX GPU-Accelerated Algebraic Multigrid Solver
New1. Overview
In simulation software the cost item most easily underestimated is the linear solve. The pressure equation in CFD, the stiffness equation in finite element analysis, and the pressure equation in reservoir simulation all reduce to large sparse linear systems. Once the mesh reaches tens of millions of unknowns, this step frequently consumes half of the total solve time. Algebraic multigrid (AMG) is the workhorse algorithm for this class of problem, and its value is that it pushes the cost from near linear to essentially independent of mesh size, at the price of a complicated and hard-to-auto-tune setup phase.
The difficulty with AMG on GPUs is structural. Classical AMG implementations are designed around the coarse-grained parallelism of CPU multicore systems, while GPUs demand fine-grained parallelism, explicit data movement, and high-throughput sparse matrix-vector products. Porting AMG line by line is hard because the multigrid hierarchy itself is a graph algorithm involving strength-of-connection, aggregation, and interpolation weight construction, whereas GPUs excel at dense regular arithmetic. AMGX addresses this directly by rebuilding aggregation around parallel graph matching, smoothers around parallel graph coloring, and the storage layout around GPU sparse formats.
This entry matters to physical AI engineering teams on two levels. The first is immediate usability: AMGX is open source under a BSD license, is backed by a peer-reviewed library paper, and can be embedded directly in a CFD or finite element code or plugged into an existing solver stack through the PCAMGX interface in PETSc. The second is methodological: it demonstrates what it takes to reorganize a classical CPU numerical algorithm into a GPU-native form, and the cost and benefit boundaries of that transformation map directly onto the general question discussed in the guide on classical solver cost boundaries.
2. How It Works
The solver composition starts with the library core. AMGX exposes a simple C API that abstracts parallelism and scale across a single or multiple GPUs, with the parallel layer carried by a user-provided MPI implementation. The library contains a flexible solver composition system that allows complex nested solvers and preconditioners to be constructed, and it is explicitly suited to implicit unstructured methods.
Algorithmically the library offers two AMG families: the classical Ruge-Stuben route and the unsmoothed aggregation route, each with different selector and interpolation strategies. A variety of smoothers and preconditioners are included, covering block-Jacobi, Gauss-Seidel, and incomplete LU factorization, and the standard preconditioned Krylov subspace iterative methods can be combined with any of the available multigrid methods or simpler preconditioners.
Precision support spans fp32, fp64, and mixed-precision solve, with complex datatype support currently limited. Both scalar and coupled block systems are supported.
The parallel strategy is the design core. Parallelism in the aggregation scheme exploits parallel graph matching techniques, while smoothers and preconditioners rely on parallel graph coloring algorithms. These two choices determine how the hierarchy is constructed and how each level is smoothed on the GPU, and they explain why a line-by-line port does not work.
Configuration is exposed through a JSON format solver configuration with many exposed parameters, covering AMG method choice, cycle type, level count, coarse selector, pre-sweeps and post-sweeps, strength threshold, aggressive coarsening levels, coarse solve, and smoother selection. The modular structure makes it straightforward to implement custom methods.
Integration paths are documented. The README lists bindings including a PETSc wrapper plugin contributed by a third party in the barbagroup repository, a Julia binding in AMGX.jl, and a Python binding in pyamgx maintained by a community user, with detailed interface description in the API reference PDF shipped in the repository.
The positioning inside PETSc is documented on the official PCAMGX manual page, which lists the full option set. The AMG algorithm is selected with CLASSICAL or AGREGATION, the cycle type with V, W, F, or CG, and the coarse selector with SIZE_2, SIZE_4, SIZE_8, MULTI_PAIRWISE, PMIS, or HMIS. The coarse solver supports DENSE_LU_SOLVER and NOSOLVER, and available smoothers include PCG, PCGF, PBICGSTAB, GMRES, FGMRES, JACOBI_L1, BLOCK_JACOBI, GS, MULTICOLOR_GS, MULTICOLOR_ILU, MULTICOLOR_DILU, CHEBYSHEV_POLY, and NOSOLVER. One usage note in the manual deserves attention: the implementation accepts host or device pointers, but good performance requires that the Krylov solver be GPU accelerated as well, so that data is not transferred frequently between host and device.
Build requirements are explicit. The project needs CMake and the CUDA Toolkit, and the distributed version also needs an MPI implementation such as OpenMPI on Linux or MPICH on Windows, with a compiler supporting C++11 and an NVIDIA GPU with compute capability of at least 3.0. Within the NVIDIA HPC SDK, CMake 3.22 or newer is required.
3. Scope and Assumptions
The boundaries are worth checking one by one against your own setup.
An NVIDIA GPU and the CUDA toolchain are required. AMGX is a CUDA-native implementation with no CPU fallback path. If a workload must run across several kinds of hardware, or primarily on CPU clusters, making AMGX the only linear solver core carries migration cost.
The hardware floor has moved with the versions. Release v2.5.0 raised the minimum CUDA version from 10.0 to 12.0 and removed support for SM20, SM35, SM52, and SM60. The new minimum is Volta at SM70 and above, with SM75, SM80, SM86, SM89, SM90, SM100, and SM120 supported, tested on Hopper and Blackwell including the RTX Pro 6000. Older datacenter cards are no longer covered by that release.
Pure GPU solving is not always globally optimal. The setup phase of AMG may not benefit on GPU, and the real gain usually appears when the same or a neighbouring matrix is solved repeatedly, as in implicit time stepping. For a single solve on a special matrix structure the advantage narrows. Total simulation time also depends on whether the outer physics kernel also moves to GPU, which is precisely the problem addressed by GPU programming frameworks and JAX-based fluid frameworks.
The matrix type matters. The official positioning is that the library is well suited for implicit unstructured methods, which bounds where its strength lies. Structured grids and regularly assembled problems are better served by dedicated multigrid or direct solvers.
Tuning effort is real. The configuration exposes many interacting parameters, and the AMG setup phase is sensitive to the combination of smoother, interpolation, selector, and threshold. The library offers grid statistics output and verbose logging for diagnosis, but automatic tuning of convergence behavior still consumes human time.
The precision option changed in v2.5.0. Mixed-precision support was removed in that release together with the DISABLE_MIXED_PRECISION define, so existing workflows that depend on it need a migration assessment.
4. Getting Started
Establish a CPU baseline first. Run AMG or a direct solve on the same matrix on CPU and record iteration count, setup time, solve time, and the residual curve. Without a baseline there is no way to attribute the benefit of the GPU version.
Decide the integration route. There are two. One is to use the AMGX C API directly and assemble inside your own solver code, which offers maximum control but requires you to handle matrix format conversion and device memory management. The other is to go through the PCAMGX interface in PETSc, which requires smaller changes, provides a complete option set, and sits in a mature ecosystem at the cost of PETSc abstractions. Most projects should start with the second.
Move the matrix onto the device. In distributed settings use CUDA-aware MPI so the full matrix is not transferred between host and device every step. On the PETSc side this corresponds to combining the GPU MPI options with a GPU-accelerated Krylov solver.
Make sure the outer solver is on GPU too. This is stated explicitly in the PETSc manual: a GPU linear solver inside a CPU Krylov solver loses the acceleration to data movement.
Get running with defaults before tuning. The library ships example solvers and several JSON configurations under the src/configs directory, such as FGMRES_AGGREGATION.json. The sample matrix is provided in Matrix Market format, and by default the examples use a vector of ones as the right-hand side and a vector of zeros as the initial solution, which you replace with your own values.
Watch version and architecture matching. From v2.5.0 the standard CMAKE_CUDA_ARCHITECTURES replaces the deprecated CUDA_ARCH, the Thrust submodule dependency is replaced by the system or CUDA-bundled Thrust, the OpenMP dependency is removed entirely, and NVTX linking moves to NVTX3. The Thrust submodule arrangement changed from v2.4.0 onwards, so cloning requires the recursive flag.
Wire up CI. The ci directory in the repository provides automated testing infrastructure that can validate your build configuration in continuous integration.
5. Comparison with Alternatives
Against CPU-side hypre and PETSc native AMG. PETSc ships PCGAMG backed by hypre and PCHYPRE, both of which are the de facto CPU standards. The distinguishing feature of AMGX is that the hierarchy construction and the smoothing are rewritten around the GPU execution model. The library paper reports that the AMG algorithm achieves a two- to fivefold speedup on a single GPU against a competitive CPU implementation, with both the setup and solve phases scaling well across multiple nodes and sustaining that performance advantage. In clusters that mix CPU and GPU, both paths usually coexist.
Against direct sparse kernels such as cuSPARSE. General GPU sparse libraries provide SpMV, sparse LU, and similar primitives, whereas AMGX provides a convergence strategy. The two compose: cuSPARSE-style kernels handle coarse-level solves and some operator work inside AMGX, and the v2.4.0 change of the SpMV algorithm selection to CUSPARSE_CSRMV_ALG1 is exactly that layer being tuned.
Against simple preconditioners such as Jacobi. On small or well-structured problems a Jacobi-preconditioned CG is sufficient and extremely cheap to implement. The cost benefit of AMGX only materializes once the problem genuinely scales up.
Against an in-house GPU solver. If your team already runs a GPU linear algebra stack, a general GPU programming framework can carry an in-house path, but the AMG hierarchy construction algorithm itself requires dedicated effort, which is the central reason to reuse an existing open-source implementation.
Against the GPU porting paths of CFD codes such as OpenFOAM. Those efforts often start from the pressure-equation solver, and AMGX is a component that can be grafted in directly. In the coupled-step strategies shown in multiphysics coupling cases, the implicit coupling step is usually where linear solving dominates.
Against differentiable solver research. Recent work tries to make the linear algebra layer differentiable so gradients can propagate, which is a different objective from GPU acceleration. The AMGX C API and its GPU focus do not provide differentiability, so this route needs separate evaluation if end-to-end gradients are the goal.
6. Benchmarks and Evidence
The official paper provides the headline numbers. The AMGX library paper by Naumov and colleagues, published in the SIAM Journal on Scientific Computing in January 2015, volume 37 issue 5, pages S602 to S626, DOI 10.1137/140980260, states that the AMG algorithm achieves a two- to fivefold speedup on a single GPU against a competitive CPU implementation, and that both setup and solve phases scale well across multiple nodes while sustaining this performance advantage. The abstract also notes that solutions of large sparse linear systems arise in applications such as computational fluid dynamics and oil reservoir simulation, where matrices are often so large that large-scale distributed parallel computing is required.
The paper has twelve authors, including Naumov, Arsaev, Castonguay, Cohen, Demouth, Eaton, Layton, Markovskiy, Reguly, Sakharnykh, Sellappan, and Strzodka.
Version v2.5.0, published on 21 December 2025, added Blackwell support including B200, GB200, and the RTX Pro 6000, plus CUDA 13 support. The minimum CUDA version rose from 10.0 to 12.0 and was tested up to 13.0. SM20, SM35, SM52, and SM60 were dropped, and support was added for SM75, SM80, SM86, SM89, SM90, SM100, and SM120, tested on Hopper and Blackwell including the RTX Pro 6000. The CUDA_ARCH flag was deprecated in favor of the standard CMAKE_CUDA_ARCHITECTURES, the Thrust submodule dependency was removed in favor of the system or CUDA-bundled Thrust, the OpenMP dependency was removed entirely, and NVTX linking moved to NVTX3. Mixed-precision support was removed, and the cuSPARSE interface was consolidated onto the generic SpGEMM API with new flags such as use_cusparse_spgemm and cusparse_spgemm_alg. Performance and correctness fixes included optimizing hash_set insertion, fixing a performance defect in fill_A_kernel_1x1, fixing an MPI communication duplication bug, changing the convergence check to be relative, fixing an integer overflow in dense LU, and fixing latency hiding to use the global row count. The MPI examples gained the -cd flag for diagonal dominance checking, -om for matrix output, and -r for performance repeat runs.
Version v2.4.0, published on 25 October 2023 with the changelog dated 6 October 2023, raised the maximum CUDA version to 12.2 and added HPC SDK 23.7 support. It fixed an issue preventing parameters from being updated after configuration initialization, restructured all source files and removed the plugin feature, replaced the custom memory pool with cudaMallocAsync when USE_CUDAMALLOCASYNC is defined, changed the SpMV algorithm to CUSPARSE_CSRMV_ALG1, and enabled a single-kernel csrmv when the number of rows in the local matrix falls below three times the number of streaming multiprocessors on the target GPUs. The Thrust version was raised to 2.1.0 and locked as a submodule, a divergence check added the rel_div_tolerance parameter, and a new AMGX_matrix_check_symmetry API call was introduced. The recorded test matrix covers Ubuntu 20.04 and 22.04, NVHPC 23.7, GCC 9.4.0 and 12.1, OpenMPI 4.0.x, CUDA 11.2, 11.8, and 12.2, on A100 and H100 GPUs.
The version lineage is v2.0.0 on 21 March 2020, v2.0.1 and v2.1.0 on the same day, v2.2.0 on 6 April 2021, v2.3.0 on 29 June 2022, v2.4.0 on 25 October 2023, and v2.5.0 on 21 December 2025. The repository version file currently reads 2.5.0. The sample output in the README shows a build version of 2.0.0-public-build125 built on 7 October 2017.
Repository scale indicators: the repository was created on 4 October 2017, the default branch is main, the most recent push was 9 July 2026, with 705 stars, 202 forks, and 140 open issues.
Sample solver output indicators: the single-GPU example with a 12-row, 61-nonzero sample matrix under an aggregation AMG with FGMRES configuration starts from an initial residual of 3.464102 and reaches a final residual of 1.619840e-14 in a single iteration, with setup at 0.00100198 seconds, solve at 0.000689248 seconds, and total time at 0.00169123 seconds. The two-process multi-GPU example takes nine total iterations to a final residual of 1.651436e-13 with total time 0.0170917 seconds. The sample matrix is extremely small, so these numbers reflect call overhead and residual convergence behavior rather than production-scale performance.
7. Role in Physics-AI Workflows
One use is letting surrogate-model training and classical simulation share the same compute pool. On a GPU cluster training surrogate models, the classical solvers inside also run AMGX kernels, which avoids maintaining two separate hardware allocations for two workload types. This is the concrete form of the shared-infrastructure idea described in the physics AI learning path.
A second use is providing linear solve support for implicit coupling steps. In coupled multiphysics problems the linear system inside an implicit coupling step is typically large and structurally complex, and the aggregation AMG with block-system support of AMGX targets exactly this situation. The domain represented by the reservoir simulation case was an early AMGX battlefield.
A third use is supporting fast iteration in differentiable and surrogate modeling research. If the research direction is to replace or accelerate the linear algebra layer with neural networks, AMGX offers a validated high-performance reference implementation and a comparison baseline. The method discussion in the surrogate modeling and reduced-order model collection reads well in that context.
A fourth use is serving as the projection step in hybrid loops. One engineering pattern has a machine learning model propose a field and a linear solver complete the constrained projection. Such hybrid loops are sensitive to absolute solver speed, so a GPU-native implementation pays off more directly there. Topology and geometry optimization, which repeatedly solve the same family of operators, belongs to this class, as discussed in the topology optimization case.
A fifth use is providing a comparison coordinate for numerical infrastructure selection. When the question is whether to adopt a GPU-specific library or a general CPU library, AMGX gives a candidate with a public paper, an open-source license, and a version record, whose setup and solve timings, iteration counts, and residual curves can all serve as the reference yardstick for an in-house solution.
A sixth use is as a vehicle for teaching and algorithmic research. The GPU implementation of AMG hierarchy construction and smoothing is a suitable object for studying graph algorithm parallelization, sparse matrix formats, and memory hierarchy optimization. The corresponding hardware programming framework reference is the NVIDIA Warp tool entry.
8. Limitations and Failure Modes
A CPU-dominant environment does not fit. Without a CUDA device or with device capability below the floor, AMGX cannot serve as the main path.
Older GPU architectures fall outside the newest release. Release v2.5.0 removed SM60 and earlier support with a minimum of SM70, so legacy clusters need to stay on v2.4.0 or earlier.
Workflows depending on mixed precision break on v2.5.0. Mixed-precision support was removed, so existing code that uses it must migrate to fp32 or fp64 or remain on an older release.
Scenarios where setup cannot be amortized see limited benefit. For a single solve, or for matrices with special structure such as strong anisotropy, setup cost dominates and the GPU speedup shrinks.
A CPU outer solver cancels the gain. Data movement between host and device can consume the acceleration that the GPU linear algebra itself delivers.
The Krylov phase after AMG is not a GPU strength. When the residual is already small and iteration counts are few, kernel launch and synchronization overhead take a larger share.
Tuning failures look like stalling. Poor AMG convergence usually stems from a smoother and interpolation choice that mismatches the problem rather than from a numerical implementation defect, and without diagnostic experience it is easy to misattribute this to a solver bug.
Platform coverage is uneven. The v2.4.0 changelog notes that although AMGX supports building on Windows, testing on Windows is not complete, so production deployments need their own verification.
Upstream iteration pace is moderate. The gap from v2.3.0 to v2.4.0 is about seventeen months and from v2.4.0 to v2.5.0 about fourteen months, which matters when a specific capability is required.
9. Version and Source Notes
On the license. The repository root has no file named LICENSE or COPYING; the license file lives in a LICENSES subdirectory as BSD-3-Clause.txt. Its content is the BSD 3-Clause terms with a copyright line reading Copyright (c) 2017 - 2024 NVIDIA CORPORATION. All rights reserved. The GitHub API reports no license for this repository because the file name and location do not match GitHub auto-detection conventions, which is a metadata shape issue and does not affect the validity of the license itself. The license field of this entry has been rewritten to match the license file verbatim.
On repository metadata. The repository is NVIDIA/AMGX, described as a distributed multigrid linear solver library on GPU, created on 4 October 2017, with the most recent push on 9 July 2026, 705 stars, 202 forks, 140 open issues, and main as the default branch.
On version numbers. The ReleaseVersion.txt file reads 2.5.0, matching the latest release tag v2.5.0 from 21 December 2025. The CHANGELOG file still stops at v2.4.0 dated 6 October 2023, while the v2.5.0 changes are recorded in the GitHub release notes. Both exist, so a citation should say which source it uses.
On the paper. The AMGX library paper appeared in the SIAM Journal on Scientific Computing in January 2015, volume 37 issue 5, pages S602 to S626, DOI 10.1137/140980260. The README also links three documents: an industrial application introduction, a v1.0 note on enabling reservoir simulation with classical AMG, and that SIAM paper.
On build and dependency requirements. The README states the dependencies and requirements: CMake, the CUDA Toolkit, an MPI implementation for the distributed version, a compiler with C++11 support, and an NVIDIA GPU with compute capability of at least 3.0. Build artifacts are a shared library and a static library, libamgxsh.so or amgxsh.dll and libamgx.a or amgx.lib, plus example binaries from the examples directory. The sample matrix uses Matrix Market format.
On integration paths. The README lists three third-party bindings: the PETSc wrapper plugin barbagroup/AmgXWrapper, the Julia binding JuliaGPU/AMGX.jl, and the Python binding shwina/pyamgx. All three are community maintained and are listed under a plugins and bindings heading, so citations should distinguish officially maintained parts from community ones.
On the PETSc version. The official PCAMGX manual page is labeled version 3.26.1 with a page update date of 11 November 2026, corresponding to commit v3.26.1-10-g7db7fb0d8eda. The page also lists interfaces such as PCAmgXGetResources.
10. References
- Naumov M., Arsaev A., Castonguay P., Cohen E., Demouth A., Eaton E., et al., "AmgX: A Library for GPU Accelerated Algebraic Multigrid and Preconditioned Iterative Methods", SIAM Journal on Scientific Computing, Vol. 37, No. 5, January 2015, pp. S602-S626, DOI 10.1137/140980260
- NVIDIA/AMGX code repository, including the README, CHANGELOG, ReleaseVersion.txt, LICENSES/BSD-3-Clause.txt, solver configurations in src/configs, the ci test infrastructure, and the interface document doc/AMGX_Reference.pdf, https://github.com/NVIDIA/AMGX
- NVIDIA/AMGX v2.5.0 release notes dated 21 December 2025, covering Blackwell and CUDA 13 support, the raised CUDA 12.0 minimum, architecture support changes, build system changes, and cuSPARSE interface updates, https://github.com/NVIDIA/AMGX/releases/tag/v2.5.0
- NVIDIA/AMGX v2.4.0 release notes and CHANGELOG entry from October 2023, including CUDA 12.2 and HPC SDK 23.7 support and the test configuration matrix covering A100 and H100 with CUDA 11.2, 11.8, and 12.2 on Ubuntu 20.04 and 22.04 with OpenMPI 4.0.x, https://github.com/NVIDIA/AMGX/blob/main/CHANGELOG
- PETSc official manual page for PCAMGX, version 3.26.1, listing all -pc_amgx_* options, the smoother and selector value ranges, and the note about requiring a GPU-accelerated outer Krylov solver, https://petsc.org/release/manualpages/PC/PCAMGX/index.html
- The LICENSES/BSD-3-Clause.txt file in the NVIDIA/AMGX repository, with the copyright line Copyright (c) 2017 - 2024 NVIDIA CORPORATION. All rights reserved. and the three BSD redistribution conditions, https://github.com/NVIDIA/AMGX/blob/main/LICENSES/BSD-3-Clause.txt
- NVIDIA developer blog article "AmgX: Multi-Grid Accelerated Linear Solvers for Industrial Applications" together with the "AmgX V1.0: Enabling Reservoir Simulation with Classical AMG" case note, both linked from the further reading section of the AMGX README
- Community-maintained PETSc wrapper plugin barbagroup/AmgXWrapper, one of the three bindings listed in the README, https://github.com/barbagroup/AmgXWrapper
- Community-maintained Julia binding JuliaGPU/AMGX.jl, one of the three bindings listed in the README, https://github.com/JuliaGPU/AMGX.jl
- Community-maintained Python binding shwina/pyamgx, one of the three bindings listed in the README, https://github.com/shwina/pyamgx
- The hypre repository of parallel solvers for sparse linear systems featuring multigrid methods, one of the CPU-side AMG standards used by PETSc through the PCGAMG and PCHYPRE interfaces, https://github.com/hypre-space/hypre