ClimateLearn: Benchmarking Machine Learning for Weather and Climate

New
License: MIT · Updated 2026-11-11
My rating

1. Overview

ClimateLearn is an open-source PyTorch library for data-driven weather and climate modeling, developed and maintained by the Machine Intelligence Group at UCLA. It was released alongside its paper in July 2023 and is distributed under the MIT licence. Its stated purpose is to standardise datasets, model implementations and evaluation protocols for a domain where, in the authors' framing, researchers have been using inconsistent or underspecified data, training setups and evaluation.

That framing is the reason the library exists as a package rather than as a set of scripts. WeatherBench, ClimateBench and FloodNet each propose datasets and baselines for particular tasks, but no single artefact previously covered data processing, model implementations and evaluation across several tasks at once. ClimateLearn's contribution is coverage and consistency, and understanding that matters for anyone deciding whether to adopt it: the value is in the plumbing being standard, not in any single model inside it being state of the art.

This guide is written for engineers and researchers who need to run a climate-machine-learning experiment and want the result to be comparable with published work. It covers what the library actually contains, what its benchmark numbers mean and where they come from, and the decisions that the library makes on your behalf, such as normalisation scheme, latitude weighting and split boundaries.

The intended reader has working knowledge of PyTorch and of gridded geophysical data. No prior familiarity with the climate modelling literature is assumed, although the tasks section explains the three task families the library is built around, since the task definition determines which dataset, split and metric combination is appropriate.

2. How It Works

The library is organised around three task families. Weather forecasting predicts a field at time t plus delta t given the current state and optionally preceding states, with tensors shaped channel by height by width. Downscaling maps a coarse field to a finer one, correcting climate model bias and increasing resolution, so the output grid is larger than the input. Climate projection predicts long-term distributions such as annual mean surface temperature under forcing levels such as atmospheric carbon dioxide or methane.

Data access is layered. ERA5 comes either from WeatherBench as a smaller pre-processed version or directly from Copernicus with an API key. The CMIP6 collection is accessed through WeatherBench or through the Earth System Grid Federation, and ClimateLearn supports the MPI-ESM1.2-HR model specifically because its variables resemble ERA5. PRISM provides observed atmospheric variables over the contiguous United States, regridded from its native four-kilometre daily grid to 0.75 degrees. An Extreme-ERA5 subset is derived locally by thresholding variables against critical values to evaluate behaviour on heat waves and cold snaps.

Processing converts NetCDF to NumPy archives in shards, with configurable training, validation and test year boundaries. Loading happens through PyTorch data modules, and the legal task arguments are direct forecasting, iterative forecasting, continuous forecasting and downscaling, with a separate module for climate projection data.

Models are loaded by name from a data module that determines channel counts. Baselines include climatology and persistence for forecasting, nearest and bilinear interpolation for downscaling, and linear regression across several tasks. Deep learning architectures include ResNet, U-Net and Vision Transformer, and a literature benchmark model from Rasp and Thuerey can be loaded in one line. Optimizers and schedulers can be passed as names with keyword arguments, or instantiated directly and mixed.

Metrics are grouped by task: deterministic metrics such as mean squared error, mean absolute error, root mean squared error, anomaly correlation coefficient, Pearson correlation, mean bias and normalized root mean squared error in both spatial and global-mean variants; probabilistic metrics for continuous ranked probability score and spread-skill ratio. Latitude-weighted variants exist for the spatial metrics, because grid cells near the equator cover less area than cells at high latitudes. Visualizations render per-pixel bias for a specific prediction and mean bias across the test set, and for probabilistic output the rank histogram.

3. Scope and Assumptions

The library assumes PyTorch as the model interface, Python 3.8 or newer, and a separate installation of xESMF, whose ESMpy dependency is only distributed through Conda. That last point is a practical friction point that will affect environment setup on a machine without Conda available.

Data-wise, the scope is defined by three access routes. WeatherBench versions of ERA5 come at 1.40625, 2.8125 and 5.625 degrees with hourly coverage from 1978 to 2018 and pressure levels of 50, 250, 500, 600, 700, 850 and 925 hPa. The CMIP6 route covers 1850 to 2015 at 2.8125 and 5.625 degrees. The PRISM route covers 1981 to the present at daily intervals, regridded to 0.75 degrees. Data beyond those routes is not part of the library's contract.

Several processing choices are fixed rather than configurable. The documentation states plainly that all data is currently normalized to a standard normal distribution, and notes that this is not the best transform for every variable, offering precipitation as the counterexample, with different transforms listed as future work. Splits are given by explicit year boundaries passed to the processing function. The bias metric for downscaling is defined as the difference between spatial means of ground truth and prediction, and downscaling targets from PRISM are padded with zeros to a common grid shape, with metrics masked over the ocean.

A further assumption concerns model capacity and hardware. The benchmark models are small convolutional and transformer backbones, and the paper's own framing of what remains to be built includes ensembles for uncertainty quantification, a hub of pretrained neural network models for weather and climate, and hybrid baselines that combine physical models with machine learning. Those are stated intentions rather than shipped features.

4. Getting Started

Installation is pip install climate-learn, with xESMF installed separately from conda-forge. The repository ships a quickstart notebook intended for Google Colab that walks from data download through training and prediction visualisation, and the project also maintains a ReadTheDocs site with a user guide and a developer guide.

A sensible first workflow follows the library's own task structure. Pick the task first, because the task determines the dataset, the split and the metric family. For forecasting, download the variables needed from WeatherBench, convert them with the NetCDF-to-NumPy converter specifying the training, validation and test year boundaries and the number of shards, then build a data module pointing at the processed directory.

For downscaling, the same sequence applies with an input directory and an output directory, and note that PRISM targets require masking over the ocean, which the library's metric computation handles but a custom loss function would have to reproduce.

The model load call takes the data module, which determines the input and output channel counts, plus an architecture name. Supplying optimizers and schedulers by name with keyword arguments is the shortest path; instantiating them directly is available when a specific optimizer is needed, and the two styles can be mixed.

5. Comparison with Alternatives

Against WeatherBench, the relationship is one of scope rather than rivalry. WeatherBench supplies the data; ClimateLearn adds the processing pipeline, the model zoo, the metric implementation and the visualisation layer on top of the same sources. A team already standardised on WeatherBench data can adopt ClimateLearn for training and evaluation without changing where data comes from.

Against ClimateBench, the difference is task coverage and metrics. ClimateBench focuses on climate projection evaluation and contributes the normalized spatial and global root mean squared error metrics that ClimateLearn also implements for that task. ClimateLearn is broader in task families while borrowing ClimateBench's metric definitions where the tasks coincide.

Against raw PyTorch plus xarray, the difference is reproducibility of the pipeline. Writing the pipeline by hand is entirely feasible and gives full control. ClimateLearn's contribution is that the split boundaries, the normalisation, the weighting and the metric definitions are fixed and identical across users, which is what makes a leaderboard number meaningful.

Against the operational forecasting stack, no comparison is intended. The paper benchmarks against IFS, the European Centre for Medium-Range Weather Forecasts integrated forecast system, as a physics-based reference line, and reports that the deep learning methods beat climatology and persistence while trailing IFS. That gap is the honest framing of what a data-driven model achieves on these tasks.

Against the neural weather model literature, ClimateLearn provides the evaluation scaffolding for the smaller architectures. Models such as Pangu-Weather, FourCastNet and GraphCast operate at different resolutions and problem scales; the library's stated plan to integrate a hub of pretrained models and fine-tuning is precisely the bridge toward that literature.

6. Benchmarks and Evidence

The paper's forecasting benchmark uses ERA5 at 5.625 degrees, a 32 by 64 grid, forecasting geopotential at 500 hPa, temperature at 850 hPa and two-metre temperature at lead times of six hours and one, three, five and ten days. Inputs comprise 49 variables, giving 141 input channels after three timesteps of history are stacked, trained from 1979 to 2015, validated in 2016 and tested in 2017 and 2018, with AdamW, a learning rate of five times ten to the minus four and fifty epochs with early stopping at patience five.

Results are reported as latitude-weighted RMSE and anomaly correlation coefficient. The qualitative findings are the transferable ones: all methods degrade as lead time grows; the deep learning models substantially outperform climatology and persistence while remaining behind IFS; and ResNet is the strongest of the three deep learning architectures on most tasks in both metrics. The authors attribute U-Net's relative weakness to its tendency to perform better on high-resolution data and Vision Transformer's to overfitting when trained from scratch.

The extreme-weather experiment reports that deep learning models and persistence both perform better on Extreme-ERA5 than on the full ERA5 test set at every lead time, while climatology performs worse, which is consistent with predicting the mean being a poor strategy for outliers. The stated interpretation is that the marginal distribution of the subset is extreme while the conditional distribution being modelled is not.

The distribution-shift experiment trains on ERA5 and on CMIP6 and evaluates across the two. Same-dataset evaluation is better in every case, but cross-dataset performance trails by a small margin, which the authors read as robustness. The shift test drops four variables unavailable in CMIP6 and uses six-hourly data with years 1979 to 2010, 2011 to 2012 and 2013 to 2014 for train, validation and test.

The downscaling benchmark covers two settings: 5.625 degree ERA5 to 2.8125 degree ERA5, and 2.8125 degree ERA5 over the contiguous United States to 0.75 degree PRISM at daily intervals. On the ERA5-to-ERA5 setting the Pearson correlation was 1.0 for every method. Deep learning models cut RMSE well below both interpolation baselines but tend to overestimate, producing negative mean bias. In the cross-dataset setting all baselines degrade, with deep learning models retaining lower RMSE and higher Pearson correlation than interpolation.

7. Role in Physics-AI Workflows

The primary role is establishing the evaluation contract for climate machine learning. When results are reported with a documented dataset, split, normalisation, weighting and metric, they can be compared across teams. A field where independent implementations disagree about these choices produces results that cannot be aggregated, and the library's contribution is precisely to remove that ambiguity.

A second role is the reproducible baseline. The baseline set is small but diagnostic: climatology and persistence bracket what trivial predictors achieve, interpolation brackets what resampling achieves, and linear regression brackets what a linear model achieves. A new deep learning architecture that fails to beat these brackets has not earned its complexity, and the library makes that check cheap.

A third role is as a scaffold for hybrid methods. The paper explicitly names physics-informed neural networks and other hybrid baselines that amalgamate physical models with machine learning as planned work. For physics AI practitioners, this is the natural place to add a conservation-respecting loss or a physics-based constraint to an otherwise standard architecture and measure the gain under a fixed protocol.

A fourth role is teaching and onboarding. The quickstart notebook and the structured user guide lower the barrier for a domain scientist who knows climate data but not deep learning infrastructure, and for an ML practitioner who needs to understand the meteorological conventions around channels, levels and leads.

8. Limitations and Failure Modes

The most consequential limitation is the fixed normalisation. Every variable is standardised to a standard normal distribution, and the documentation acknowledges that this is a poor choice for quantities such as precipitation, which is neither symmetric nor approximately Gaussian. Models trained this way optimise the wrong objective for heavy-tailed variables, and the fix is not currently available in the library.

The second is the modest scale of the shipped model zoo. Three deep learning architectures plus one literature benchmark, with a note that the goal is to implement as many architectures as possible for fair comparison, leaves the leaderboard coverage thin relative to the literature. The planned pretrained model hub and ensemble support are the natural remedies and were not present at release.

The third is coverage of data sources. Regional datasets beyond PRISM, and additional reanalysis and observation products, are outside the library. Anyone working outside the global ERA5 grid or the contiguous United States has to extend the download and processing layer themselves.

The fourth is that this is a benchmarking library, not a forecasting production system. It offers no operational pipeline, no data assimilation, no latency budget and no reliability guarantee for a deployed service. Model error grows with lead time in the benchmark itself, and the iterative-forecasting approach accumulates error exponentially at longer horizons, which the paper documents and attributes to prior work as well.

The fifth is project maturity. The package metadata classifies the project at pre-alpha development status, and the documentation states that the project is under active development and that the API may undergo extensive changes. Version 1.0.0 was published on PyPI in July 2023 with two earlier pre-releases from February 2023, and the public release list has not extended beyond that. Code written against the API should expect to accommodate change.

9. Version and Source Notes

The published release history of the repository contains a single entry, version 1.0.0, dated 11 July 2023, whose notes identify it as the code used for the accompanying paper and list the contributors. On PyPI, three versions exist: 0.0.1 and 0.0.2 both uploaded on 21 February 2023, and 1.0.0 uploaded on 6 July 2023. The discrepancy between the 6 July PyPI upload and the 11 July GitHub release is a five-day gap between packaging and tagging and is recorded here rather than smoothed over.

Package metadata declares Python 3.8 or newer, and the license classifier is MIT. The repository licence file and the metadata agree. The documentation site identifies the version as 1.0.0 and shows a revision identifier, indicating the docs are built from the tagged release rather than continuously from the main branch.

Two content caveats belong with the benchmarks. First, the paper's forecasting comparison places the deep learning methods behind IFS, the operational physics-based forecast, and this ordering should be reported alongside any use of the library's numbers, since quoting only the deep learning rows would misrepresent the state of the art on these tasks. Second, the extreme-weather subset is derived by thresholding individual climate variables locally, and the authors note that flooding, typically caused by heavy precipitation over short periods, does not qualify under the categories they used; a dataset aimed at flood evaluation would need a different construction.

Provenance for this guide: version and packaging facts come from the GitHub release entry and PyPI metadata; library scope, task definitions, model and metric lists come from the official documentation; all benchmark numbers come from the ClimateLearn paper and are the authors' own results.

10. References

  1. ClimateLearn project README, GitHub repository aditya-grover/climate-learn, library scope, installation requirements, quickstart notebook and citation guidance.
  1. ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling, arXiv 2307.01909, submitted 4 July 2023, dataset and model pipeline description and benchmark experiments.
  1. ClimateLearn v1.0.0 release notes, 11 July 2023, the code used for the paper and the contributor list.
  1. PyPI package metadata for climate-learn 1.0.0, upload date, Python requirement, MIT license classifier and development status.
  1. ClimateLearn documentation, user guide section on tasks and datasets, task definitions, ERA5, CMIP6 and PRISM access routes, data processing and loading.
  1. ClimateLearn documentation, user guide section on models, baseline and deep learning architecture table, model loading and customization options.
  1. ClimateLearn documentation, user guide section on metrics, deterministic and probabilistic metric lists and their source references.
  1. ClimateLearn documentation, user guide section on visualizations, per-prediction bias and mean bias rendering.
  1. Rasp et al., ClimateBench: A multi-resolution, multi-physical benchmark for the uncertainty in climate projections, arXiv 2205.00865, source of the normalized root mean squared error metrics.
  1. WeatherBench, WeatherBench2 and related benchmark documentation, the dataset sources and baselines ClimateLearn builds on.
  1. Rasp and Thuerey, Skillful precipitation nowcasting using deep generative models of radar, 2020, the literature benchmark model loadable by name in the model zoo.
  1. ClimateLearn announcement blog post by the UCLA Machine Intelligence Group, 2023, the project's stated motivation and future plans.
Tags: