← Back to glossary

Mixed-Precision Training AMP

Method Also:Half-Precision Training / FP16 AMP Updated 2026-11-04
Definition

A training method where different layers of the network compute and store in FP16, BF16 or FP32, with operators prone to overflow automatically promoted to FP32. It nearly doubles training throughput and saves memory with minimal accuracy loss.

What it is

Mixed-precision training (AMP) means mixing float precisions within one training run. The main path (forward, backward, weight updates) uses FP16 or BF16, while a small number of numerically sensitive or overflow-prone operators - reductions, gradient computation, large magnitude accumulations - keep FP32 computation.

Frameworks handle this automatically: during the forward pass they record which operators risk overflow, then rescale gradients during backpropagation (loss scaling) to prevent underflow and unscale before the weight update. In practice the user just enables a flag.

Key points

  • BF16 versus FP16: FP16 has a narrow dynamic range (about 65504 max), so gradients

underflow easily and need loss scaling to compensate; BF16 has a range close to FP32, usually needs no scaling and is more stable, but requires newer hardware support. BF16 is the easier choice for foundation-model-scale training.

  • The memory saving is a means, not the end: activations stored in half precision allow a

much larger batch size. For neural operators, which need large sampling batches, that batch increase often decides convergence quality directly.

  • Per-operator blacklists matter: certain physics operators (division, square roots,

conservation terms with heavy accumulation) lose accuracy badly in half precision. Custom operators should explicitly declare FP32, otherwise the model may "finish training but not converge".

  • AMP for training and quantization for inference are different things: AMP targets training

throughput and stability, quantization targets deployment size and latency. They have different goals and should not be conflated.

Common pitfalls / boundaries

  • Higher precision does not mean lossless. Physics constraints, especially stiff terms carrying

physical units, may lose significant digits in half precision and need empirical verification rather than silent acceptance.

  • The loss-scaling factor is not better when larger. Too large underflows gradients late in

backpropagation, and both too large and too small show up as anomalous loss early in training.

  • Some older GPUs or operators have no half-precision implementation and silently fall back to

FP32, yielding far less benefit than expected. Confirm throughput by measurement.

  • Mixed precision interacts with gradient clipping and normalization; changing one requires

re-verifying the others.

Related terms

Related resources