AirfRANS High-Fidelity Airfoil Surrogate Benchmark Case
New1. Overview
The hard part of surrogate evaluation is not training a model, it is making numbers from different teams comparable. Training split, test split and error normalisation decide whether two papers can be placed side by side, and any difference in those three quietly voids the comparison. AirfRANS fixes all three at once. It is built from 1000 high-fidelity steady RANS simulations over NACA 4-digit and 5-digit series airfoils, and it defines four machine-learning tasks over that single corpus: full data, scarce data, Reynolds extrapolation and angle-of-attack extrapolation. It also introduces surface stress metrics and boundary-layer visualisation, which together make the point that a field that looks right and a coefficient that is usable are different achievements.
This entry is a case study focused on the benchmark design rather than on any single model's score. It explains the settings that produced the data, the solver, turbulence model, mesh and near-wall resolution; what each of the four tasks is actually probing; how the metrics should be read; and how several unverifiable claims in an earlier catalogue description here should be corrected. The audience is algorithm engineers who need a trustworthy aerodynamic evaluation protocol, aerodynamics and certification teams checking the claim that surrogates can displace part of a wind-tunnel programme, and generalist operator-learning researchers who want a clean two-dimensional turbulent benchmark. Method selection belongs to the neural operator selection guide, and cost arithmetic belongs to the classical solver cost guide.
Scope comes first. The source is a NeurIPS 2022 Datasets and Benchmarks paper together with its public data, released under ODbL v1.0. The paper reports results for the four baselines it proposes; third-party methods are not enumerated here because their splits and metrics are not aligned. This entry therefore separates benchmark design facts from interpretation, and everything was checked on 11 November 2026.
2. How It Works
The data draws on two lines. The geometry comes from early NACA airfoil work: the 4-digit and 5-digit series are fully parameterised and generate a broad shape family quickly. The 4-digit notation is MPXX, with M the maximum camber in percent, P the chordwise position of maximum camber in tenths of a chord, and XX the maximum thickness in percent. Sampling takes M uniformly over 0 to 7, P uniformly over 0 to 7 with any value below 1.5 set to zero to remove geometries whose maximum camber sits too close to the leading edge, and XX over 5 to 20. The 5-digit notation is LPQXX, where L and P define camber, Q takes 0 or 1 to distinguish single from double curvature, and XX is again thickness, with ranges of 0 to 4 for L, 3 to 8 for P, the set {0, 1} for Q and 5 to 20 for XX. The meshes come from the two-dimensional case library of the NASA Langley Turbulence Modeling Resource, which is the material basis for calling the dataset high-fidelity.
The physical setting is two-dimensional incompressible steady RANS at sea level and 298.15 K, with Reynolds number between 2 and 6 million, a Mach number below 0.3 that justifies the incompressible assumption, and a free stream above 30 m/s. The turbulence model is k-omega SST, chosen because it is well suited to separation and to force prediction. The angle of attack spans minus 5 to 15 degrees: the lower bound makes the lift coefficient of a cambered airfoil roughly zero, and the upper bound avoids stall and unsteadiness in the airfoil wake.
Numerics are handled by OpenFOAM v2112, with blockMesh generating multi-block hexahedral C-grids whose geometry references the NASA NACA 0012 and NACA 4412 grid designs, the far-field boundary sitting about 200 chords from the airfoil. Each mesh holds between 250,000 and 300,000 cells, the first cell at the surface is 2 micrometres tall and the worst-case wall distance is about one, a low-Reynolds strategy that resolves the boundary layer to the wall so that friction drag is computed rather than modelled. The solver is the steady RANS simpleFOAM with SIMPLEC pressure-velocity coupling. One simulation takes about 25 minutes on 16 CPU cores of an AMD Ryzen Threadripper 3960X, and the full dataset cost roughly 20 days of machine time.
Delivered products are split by boundary. The interior file is cropped to a box spanning minus 2 to 4, minus 1.5 to 1.5 and 0 to 1 and sliced at z equals 0.5 to become two-dimensional, carrying pressure, velocity, turbulent viscosity and a distance function. The airfoil boundary file carries pressure, velocity, turbulent viscosity and normals, and the outer boundary file carries pressure, velocity and turbulent viscosity. A raw variant preserving native OpenFOAM output is offered alongside the preprocessed one for users who need the full solver output.
3. Scope and Assumptions
The first assumption is two-dimensional, steady and incompressible. An airfoil section problem differs in kind from three-dimensional unsteady flow over a wing or a whole vehicle, so a two-dimensional steady result does not transfer directly. The second is subsonic flow over a fixed Reynolds range: the sub-0.3 Mach assumption removes shock and transonic effects, and the band from 2 to 6 million corresponds to particular chord and speed combinations. The 15 degree angle-of-attack ceiling is a deliberate physical boundary, so no data exists near stall and model behaviour beyond it is unsupported.
The third assumption is mesh and wall resolution. Between 250,000 and 300,000 cells, a 2 micrometre first layer and a wall distance near one together determine that the drag values in the dataset already carry credible friction. If a user coarsens the mesh to the ten-thousand-cell range before retraining, the resulting friction deficit makes the numbers incomparable with the benchmark. Comparability is therefore bound to the mesh specification.
The fourth assumption is the metric definition. The paper denotes relative errors of the drag and lift coefficients by rho_D and rho_L, reports them in percent, and adds Spearman rank correlation to capture ordering. Wall shear stress is derived from the velocity field rather than regressed directly. That choice matters, because field error is amplified into coefficient error through integration, so a field that looks close does not imply usable coefficients. Metrics must align with the quantity of interest downstream: lift makes rho_L the primary metric, drag makes rho_D the primary one, and the gap in difficulty between them is exactly the phenomenon the paper emphasises. The metric and surrogate conventions are set out in the surrogate modelling collection.
The fifth assumption is licensing. Both the dataset and the generation, experiment and baseline code are released under the Open Data Commons Open Database License v1.0, which permits commercial use, distribution, modification and private use while requiring source attribution, retention of licence and copyright notices, and distribution of derived databases under the same licence. That clause constrains any plan to fold the benchmark data into an internal training set and publish a model, so it should be checked line by line before commercial use.
4. Getting Started
Choose the data flavour first. The dataset ships both a preprocessed version, where cases are organised into interior, airfoil boundary and outer boundary files ready for a standard machine-learning loader, and a raw version preserving native solver output for users who want their own post-processing. The companion Python library downloads either with a single call, and a PyTorch Geometric dataset interface is available. Most users should start preprocessed and spend the compute on the model side.
State the task next. The four tasks probe different abilities: full data probes the fitting ceiling, scarce data cuts the training set to 200 cases and probes label efficiency, Reynolds extrapolation probes transfer across operating conditions, and angle-of-attack extrapolation probes transfer across attitudes. A question about accuracy within one operating condition belongs to the full task, a question about whether few labels suffice belongs to the scarce task, and a question about trusting a new operating condition belongs to an extrapolation task. Choosing the wrong task makes every accuracy number meaningless.
Build baselines next. The paper offers four: a multilayer perceptron using only node features, a graph aggregation network adding local neighbourhood information, a point network placing global features on top of a deep perceptron, and a graph U-Net that reaches from local to global neighbourhoods through multi-scale structure. Each model is trained five times to report means and standard deviations. Reproducing these four first is the most direct way to confirm that an implementation is correct, because a mismatch with the published numbers usually points at the split or the metric implementation rather than at the model.
Mind the evaluation detail. Lift and drag coefficients are integrated from the predicted field, so evaluation should look at three things together: relative error, Spearman rank correlation and boundary-layer visualisation. The paper provides the visualisation explicitly as a diagnostic, because a model that is wrong in the wake and separation region can still produce a respectable coefficient. Field and visualisation tooling can follow the conventions in PyVista and the numerics and solvers collection.
5. Comparison with Alternatives
The first comparison runs against a three-dimensional sibling. DrivAerML provides 500 parametric whole-vehicle scale-resolving cases at roughly 160 million cells each, using transient SIMPLE with an SA-sigma-sigma-DDES hybrid model, in sharp contrast to AirfRANS at two dimensions, steady, 250,000 to 300,000 cells and k-omega SST. The mesh scales differ by three orders of magnitude and the turbulence modelling tiers differ, which changes both the input representation a surrogate consumes and its memory budget. Ranking them together is invalid; comparing them is useful because it shows how problem scale changes method choice. The automotive aerodynamics case records the full picture on the vehicle side.
The second comparison is the solver. SU2 is an independent open-source solver that can cross-check airfoil cases, but AirfRANS itself uses only OpenFOAM v2112 and SU2 appears nowhere in the data generation chain. Any claim that SU2 produced this dataset can be rejected on that basis. The SU2 tool entry and the SU2 aerodynamic optimisation case describe where that solver is actually used.
The third comparison is between method families. For airfoil surrogates the Fourier neural operator and DeepONet perform strongly on regular parametric grids, with tutorials and papers available in the FNO tutorial, the DeepONet tutorial, the FNO paper and the DeepONet paper. The four baselines published with this benchmark contain no operator method, and third-party results on the benchmark vary in split and metric, so merging them into a single ranking table misleads. The reporting checklist in the neural operators practical tutorial is the right instrument for aligning them first.
The fourth comparison is reduced-order modelling. When design parameters vary smoothly and only a few integral coefficients matter, proper orthogonal decomposition is cheap, auditable and often better than a deep model that needs a large corpus. The neural operator selection guide develops the selection rule by question type.
6. Benchmarks and Evidence
The scale record is clear: 1000 RANS simulations, each defined by one airfoil, one Reynolds number and one angle of attack, each on a mesh of 250,000 to 300,000 cells drawn from the two-dimensional library of the NASA Langley Turbulence Modeling Resource, at about 25 minutes per solve on 16 cores and roughly 20 days of machine time for the full generation, published under ODbL v1.0 with code and experiments released alongside.
The task record is the most valuable part of this benchmark. The full data task trains on 800 cases and tests on 200. The scarce data task trains on 200 cases against the identical test set, which makes the pair a strictly controlled data-efficiency comparison. The Reynolds extrapolation task trains on Reynolds numbers from 3 to 5 million and tests on 2 to 3 million and 5 to 6 million. The angle-of-attack extrapolation task trains from minus 2.5 to 12.5 degrees and tests on the two outer bands from minus 5 to minus 2.5 and from 12.5 to 15 degrees. The paper does not give a validation set size in the main text nor the per-task sample counts for the extrapolation tasks, so this entry does not cite those three quantities.
The performance record follows the paper's own baselines. The paper's conclusion is that drag is markedly harder than lift. On the full data task, the graph aggregation baseline reaches a drag coefficient relative error rho_D of 4.05 plus or minus 0.70 percent and a lift coefficient relative error rho_L of 0.52 plus or minus 0.16 percent. Out of distribution, rho_D rises to 8.97 plus or minus 1.28 percent for Reynolds extrapolation and 5.59 plus or minus 1.09 percent for angle-of-attack extrapolation. The ordering result deserves more attention than the error figures, because the Spearman rank correlation of the drag coefficient falls to near zero in both extrapolation settings, meaning the model has largely lost the ability to rank candidates by drag, which is precisely the property a surrogate needs for screening. Both numbers come from the paper's tables; third-party results on this benchmark are not cited here because splits and metrics are not aligned. Reporting conventions can be checked against PDEBench and the CFD benchmark.
7. Role in Physics-AI Workflows
AirfRANS occupies the position of a trustworthy evaluation benchmark, complementing the data-factory benchmarks. A data factory answers where the corpus comes from; this benchmark answers whether the judgement itself is credible. For a team that must show management that a surrogate can displace part of a wind-tunnel programme, a fixed split and a fixed metric is the most economical argument available: one test set, one ruler, four models separated clearly.
Four usage lines follow. The first is method screening, where a candidate model runs the four tasks to see which capability it lacks, with the extrapolation tasks usually the most diagnostic. The second is boundary definition, where the scarce data task yields an accuracy-versus-label curve that answers at what annotation budget the surrogate becomes usable. The third is failure detection, where Spearman correlation enters the routine monitoring set so that a collapse in ranking capability triggers a demotion of the surrogate from screening tool to interpolator. The fourth is regression protection, where a fixed task set runs before each release to catch silent accuracy decay.
In a design-optimisation context the benchmark can also act as a cheap screening layer: a surrogate pass narrows the candidate list, and high-fidelity solves re-check only the selected few, concentrating simulation budget where the surrogate is uncertain. That mirrors the division of labour seen on the whole-vehicle side and recorded in the hybrid flow-solid surrogate case. The broader sequence sits in the physics-AI learning path.
8. Limitations and Failure Modes
The first failure class is dimension and operating condition. A two-dimensional steady result cannot answer spanwise flow over a wing, wing-body interference, propeller slipstream or unsteady buffeting, and a uniform free stream cannot answer gusts or shear. The 15 degree ceiling means no data exists near stall, so model output beyond that has no basis. Outside the Reynolds band from 2 to 6 million the boundary-layer state changes and the friction share of drag moves with it, which likewise leaves the supported range.
The second failure class is metric-to-purpose mismatch. The gap between rho_D and rho_L shows that accurate fields and accurate coefficients are not the same achievement, and a Spearman correlation near zero shows that in extrapolation even a percent-level error can accompany a complete loss of ranking. When a surrogate drives candidate ordering inside a design loop, relative error alone yields an overly optimistic conclusion.
The third failure class is a quietly changed mesh specification. Comparability is bound to 250,000 to 300,000 cells, a 2 micrometre first layer and a wall distance near one. Coarsening to the ten-thousand-cell range systematically underestimates friction, and model error then stacks on mesh error so a drop in the metric cannot be attributed. Recomputing the same geometries on a denser grid and using those as new labels likewise produces a score that is not the same benchmark.
The fourth failure class is implementation detail in the metrics. Drag coefficients are integrated from the predicted field, so mesh arrangement, normal orientation and the area weighting of integration surfaces all change the answer. Treating wall shear stress as a directly regressed quantity rather than a derived one diverges from the reference implementation. None of these raises an error; they simply make a score mysteriously worse, which is why reproduction should align with the official baselines numerically before any custom implementation is touched. Solver-side cross-checks can be run with OpenFOAM on a couple of cases, and physics-informed comparisons sit with DeepXDE and PhysicsNeMo.
9. Version and Source Notes
Record 1: arXiv:2212.07564 is titled "AirfRANS: High Fidelity Computational Fluid Dynamics Dataset for Approximating Reynolds-Averaged Navier-Stokes Solutions", authored by Florent Bonnet, Ahmed Joclyn Mazari, Paola Cinnella and Patrick Gallinari, with v1 submitted on 15 December 2022 and v3 dated 1 June 2023, published in the NeurIPS 2022 Datasets and Benchmarks Track. The catalogue year of 2022 refers to that conference publication.
Record 2: the institutional attribution needs correction. An earlier catalogue description on this site placed the related work in Orleans and Stuttgart. That does not hold. AirfRANS was produced by Extrality together with a Paris-based team, and the publicly listed affiliations include Sorbonne Universite, CNRS, ISIR, Extrality and the Criteo AI Lab. The affiliations do not change the data, but they change who a reader believes maintains the benchmark.
Record 3: the airfoil series needs correction. The earlier text said NACA 4- and 6-digit series, while the dataset actually uses the 4-digit and 5-digit series, and the string 6 digits does not appear in the paper. The report numbers need correction too. The earlier citation to NACA Report 1469 could not be verified. The reports genuinely tied to this geometry are NACA Report 460 by Jacobs, Ward and Pinkerton from 1933, which tested 78 related airfoil sections in the variable-density tunnel and established the four-digit naming scheme, and NACA Report 824 by Abbott, von Doenhoff and Stivers from 1959, which summarises airfoil data across the 4-digit, 5-digit, 6-series and 7-series families.
Record 4: the claim of a NASA official NACA 0012 public dataset does not hold. NASA did publish grid designs for NACA 0012 and NACA 4412, and AirfRANS does reference those designs, but the 1000 meshes used here come from the two-dimensional case library of the NASA Langley Turbulence Modeling Resource rather than from a product named an NACA 0012 public dataset. Conflating wind-tunnel reports with a computational case library misleads readers into expecting measured comparison data, which the dataset does not contain.
Record 5: four unverifiable performance figures are withdrawn. The earlier description gave RANS evaluation at two to four hours, a Fourier neural operator at about 10 milliseconds with drag error below 5 percent, DeepONet at about 5 milliseconds with error below 8 percent, and a CNN at about 2 milliseconds with error below 10 percent. None could be traced to a primary report in the public literature. The dataset uses only OpenFOAM v2112 and SU2 appears nowhere in the generation chain, and the four published baselines are a multilayer perceptron, a graph aggregation network, a point network and a graph U-Net, with no Fourier neural operator and no DeepONet among them. All four figures are therefore withdrawn and replaced by the paper's own baselines as cited in the benchmarks section.
Record 6: the solve time needs correction. The paper reports about 25 minutes per simulation on 16 cores and roughly 20 days of machine time for the full dataset, which does not match the two-to-four-hour order of magnitude previously stated. Record 7: the licence is confirmed as ODbL v1.0, covering the dataset, the generation code and the experiment code. Record 8: training and test counts for the four tasks follow the paper text, namely 800/200, 200/200, Reynolds training on 3 to 5 million against tests on 2 to 3 and 5 to 6 million, and angle-of-attack training on minus 2.5 to 12.5 degrees against tests on the two outer bands. Validation size and per-task extrapolation sample counts are not given in the main text and are not inferred here.
10. References
- Bonnet F., Mazari A.J., Cinnella P., Gallinari P., "AirfRANS: High Fidelity Computational Fluid Dynamics Dataset for Approximating Reynolds-Averaged Navier-Stokes Solutions", arXiv:2212.07564, v1 15 Dec 2022, v3 1 Jun 2023; NeurIPS 2022 Datasets and Benchmarks Track.
- arXiv record and full text, v3, arxiv.org/abs/2212.07564, checked 11 November 2026.
- AirfRANS open-access archive copy hosted at HAL, hal.science/hal-03806092, checked 11 November 2026.
- AirfRANS experiment and baseline code at github.com/Extrality/AirfRANS, with new simulation code at github.com/Extrality/NACA_simulation.
- AirfRANS Python library documentation, airfrans.readthedocs.io.
- Ladson C.L., Johnson W.G., "Simulation of the Two-Dimensional Incompressible Navier-Stokes Equations", NASA TM 811103 and the Turbulence Modeling Resource 2DN00 case library.
- Jacobs E.N., Ward K.E., Pinkerton R.M., "The Characteristics of 78 Related Airfoil Sections from Tests in the Variable-Density Wind Tunnel", NACA Report 460, 1933.
- Abbott I.H., von Doenhoff A.E., Stivers L.S., "Summary of Airfoil Data", NACA Report 824, 1959.
- Menter F.R., "Two-equation eddy-viscosity turbulence models for engineering applications", AIAA Journal 41, 2003, 3-22.
- Sanchez-Gonzalez A. et al., "Fourier Neural Operator for Parametric Partial Differential Equations", ICLR 2021.
- Lu L. et al., "Learning nonlinear operators via DeepONet based on the variational principle for a class of PDEs", Nature Communications 12, 2021, 4737.
- DClaw Research, "Neural Operator Selection Guide", site guide, 11 November 2026.