NVIDIA ProtoMotions

New
License: Apache License 2.0 · Updated 2026-11-11
My rating

1. Overview

ProtoMotions is an open-source, GPU-accelerated simulation and learning framework from NVIDIA Labs, released under the Apache 2.0 licence. It exists to train physically simulated digital humans and humanoid robots, and it covers the full loop from motion capture data through policy training to deployment on physical hardware. The current major version is version 3, whose design principles the project states explicitly: modularity, extensibility and scalability.

The framework's value proposition is consolidation. A humanoid learning project otherwise stitches together a physics engine, a motion dataset pipeline, a retargeting solver, an imitation learning algorithm and a deployment stack, each with its own conventions. ProtoMotions puts those pieces behind one interface, and the payoff is not convenience alone: because the same policy can be trained in one engine and evaluated in another through a single command-line switch, the engineering team can check whether a policy depends on quirks of one simulator before it meets a real robot.

This guide matters for physics AI because embodied AI is one of the areas where the physics engine itself is part of the learning problem. The agent is not a passive dataset consumer; its actions change the state distribution it will later be evaluated on, and the gap between simulation and reality is exactly the gap that decides whether the work ships. A framework that makes cross-engine validation and deployment frictionless is therefore a physics-infrastructure question, not merely a tooling preference.

The decision this guide supports is about scope and risk: which capabilities are production-ready, which are explicitly experimental, and where the framework's own release notes record known limitations. Recent releases have been unusually explicit about this, including naming what was not run on physical hardware. That candour is worth reading closely rather than skimming. All statements were checked against the repository, the release notes and the project documentation on 11 November 2026; Section 9 records the version timeline and the licensing split.

2. How It Works

The framework's central design decision is that a task is not a monolithic environment class. Instead, a task is assembled from independent components: control logic that manages task state, observation kernels that transform raw state into feature vectors, reward functions, and termination conditions. Each of these is a standalone function or class, and an experiment configuration binds them together into a complete task through descriptors that name the context paths they read and write.

The environment loop itself was restructured around explicit step phases over modular state owners and task control components, with checked boundaries between them, so that a change in one place cannot silently break another. Sensors are declared once per channel, each carrying a measurement-noise model and a state-estimation stage, which gives domain randomisation and sim-to-real transfer a principled footing rather than an ad-hoc one.

Physics backends are abstracted behind a common simulator interface, with separate implementations for NVIDIA Isaac Gym, Isaac Lab, NVIDIA Newton and MuJoCo. Switching between them is a command-line argument rather than a code change, and that is what enables the cross-engine evaluation the framework advertises. The robot asset format was standardised on the MuJoCo XML specification, with Isaac Lab converting packaged files at load time while preserving multi-axis joint structure and metadata, so a single asset file serves every backend.

Retargeting, the step that converts human motion onto a robot skeleton, is handled by the PyRoki-based optimiser from version 3 onward; earlier versions used a different solver. The pipeline is driven by a configuration system with declarative authoring of observations, rewards and terminations as pure tensor kernels.

On the learning side, the framework offers three policy families that matter for practice. A motion tracker follows a reference clip. A generative policy chooses its own movement to complete a task rather than replaying one reference. A generative pre-trained controller trains a reusable discrete latent prior and adapts it to new tasks with small parameter-efficient adapters.

The deployment path exports a policy as a single ONNX model with the observation computation baked in, so a deployment framework only has to supply raw sensor signals. The project has also prototyped a standalone C++ runtime with MuJoCo and robot backends, a safety supervisor, and a TensorRT build and qualification pipeline, and this is explicitly marked as prototyping reference code rather than a supported release artefact.

3. Scope and Assumptions

The first scope boundary is the simulator roster. The framework supports Isaac Gym, Isaac Lab, NVIDIA Newton and MuJoCo. A Genesis backend existed in earlier versions and was removed in the October 2026 refactor, so any plan that names Genesis needs revisiting. There is also a distinction between rendering and physics: Isaac Sim 5.0 and later can load Gaussian splatting backgrounds for visual fidelity, and the project states plainly that this rendered scene cannot yet be interacted with physically. Visual polish and physical correctness are separate axes here.

The second boundary is the model family. The released work covers articulated humanoids, humanoids with dexterous hands, and digital human characters including SMPL, SMPL-X, an anatomical SOMA variant and a SOMA variant. It does not cover quadrupeds, aerial vehicles or vehicles, and it does not cover manipulation of deformable objects.

The third is the asset and licensing model. The source code is Apache 2.0, but pretrained models are distributed under the NVIDIA Open Model Licence unless a model card says otherwise, and each model directory carries its own card, licence and attribution notice. Third-party software and bundled asset notices are listed separately, including attributions for the robot hardware, the imitation learning codebase, Isaac Lab and the body model assets. A commercial deployment therefore needs a licence review that covers models and assets, not just code.

The fourth boundary is what the framework does not attempt. It does not do motion capture, it does not do scene or asset authoring beyond a conversion pipeline, and it does not provide a physics engine of its own. It consumes simulators. The physics simulation primer covers the modelling choices those simulators make, and the simulation toolkit collection covers the continuum side of the field for contrast.

A fifth boundary concerns the deployment runtime. The native C++ runtime is described in the project documentation as a prototyping reference for people building their own deployment path. Treating it as a supported product with a stability guarantee would be a misreading of the project's own wording.

4. Getting Started

Installation is documented per backend, with separate requirement files for Isaac Lab, Newton, Isaac Gym and Genesis-era setups. The current documentation recommends creating a virtual environment with the uv tool and installing the package in editable mode, then installing the backend-specific requirements. The August 2026 release notes also added PyPI and uv installation support while preserving preconfigured Torch builds, which simplifies environment setup on cluster nodes.

The first training run should use the documented quickstart command, with the motion-imitation experiment configuration and an explicitly named robot. The robot selection is a command-line argument, so switching from a human character model to a robot is a parameter change rather than a code change, provided retargeted motions are prepared for the target skeleton.

The retargeting step is the one that needs preparation before training. The project provides a dedicated data preparation guide for the public AMASS motion capture dataset, for additional motion sources, and for retargeting instructions. Preparing the retargeted motions correctly is where most first-time failures occur, and the release notes are unusually specific about which artefacts changed and which did not.

A second run should be a cross-engine evaluation. Train in one simulator, then evaluate in another with a single argument change. This is the cheapest available check on whether a policy is relying on simulator-specific behaviour, and it is the check most likely to change a deployment decision. The performance engineering collection covers the profiling discipline that goes with it.

Adding a robot requires two steps and no edit to a factory: place the robot's MuJoCo XML file in the assets directory and create a robot configuration class whose direct base is the robot configuration base class. The registration mechanism scans that directory, so the file name becomes the name passed to the robot argument. Adding a simulator means implementing the simulator interface, with the MuJoCo backend offered as the smallest reference implementation. Adding a reinforcement learning algorithm is reported to take roughly fifty lines in the modular layout.

For real hardware, the project provides a deployment tutorial covering data preparation through real robot, a native deployment runtime, a live metrics dashboard, and headset-based teleoperation for the robot's hands. The Python robotics entry covers the broader tooling context for such deployments.

5. Comparison with Alternatives

The nearest sibling is the project's own MimicKit, explicitly positioned as a lightweight alternative for motion imitation learning. The division of labour is scale and scope: ProtoMotions carries multi-backend physics, multi-GPU training, generative policies and deployment, while the lighter sibling covers the imitation case. Choosing the heavier framework for a single-simulator imitation task adds dependency weight without adding capability.

Against Isaac Lab directly, the relationship is one of layer. Isaac Lab is a framework for robot learning built on Isaac Sim, offering a task-centric environment model and a large set of assets and algorithms. ProtoMotions sits above it and specialises in one problem domain: human and humanoid motion. If the goal is broad manipulation and locomotion research across many robot morphologies, Isaac Lab is the broader platform; if the goal is motion imitation from human capture onto humanoids with generation and deployment, ProtoMotions is the more specialised instrument.

Against a pipeline of separate tools, the comparison is on integration cost. A team that already has a working motion capture pipeline, a retargeting solver, an imitation learning codebase and a deployment stack may gain little from consolidating, and may in fact lose the ability to change any one component. The consolidation argument is strongest for a team starting from nothing or for one facing a simulator-specific artefact that only cross-engine testing would reveal.

Against a physics engine used directly, the framework offers the training and deployment layer but inherits the engine's physics entirely. This matters because recent releases contain a long list of per-backend corrections: joint effort readback semantics on MuJoCo, joint gain assignment by complete joint name on Newton, joint property ordering on Isaac Gym, and centre-of-mass corrections for angular pushes on Isaac Lab. The existence of that list is itself the argument for the framework, since it documents where the engines differ. The Warp entry covers the GPU kernel substrate that the Newton and MuJoCo-Warp path builds on.

For the broader picture of how learned policies fit alongside classical solvers, the classical solver cost boundary guide is the relevant companion, and PhysicsNeMo covers the physics-informed branch of the field.

6. Benchmarks and Evidence

The project's headline training figure is that a fully physically simulated character can learn motion skills from the entire public AMASS dataset, more than forty hours of human animation, within roughly twelve hours on four A100 GPUs. The multi-GPU figure is that training has been run with twenty-four A100s, thirteen thousand motions per GPU, using the BONES dataset in the SOMA skeleton format. These are the project's own statements and they describe wall-clock training time on a specific dataset and hardware configuration.

The deployment numbers come from the anatomical SOMA tracker work released in November 2026. On a shipped ten-clip pack in MuJoCo, mean body position error was 0.045 metres, and on 256 BONES-SEED clips it was 0.049 metres. The head pitch error was reported at 2.2 degrees against 18.7 degrees before the tracker was retrained with explicit head tracking. The repository also documents a data scale for the general tracking policy of roughly 142 thousand motions in the BONES-SEED set, and describes a zero-shot transfer of that policy to the Unitree G1 robot.

The validation procedure itself is worth reading because it is unusually concrete. Each shipped bundle is checked to load strictly into the model its saved configuration builds, to command joint targets inside the robot's joint range, and to step in MuJoCo; each bundle is regenerated byte for byte from its configuration file. Backend fixes were verified against the real engines where available, including Newton on CPU and Isaac Gym Preview 4 on a consumer GPU. The native deployment path reports twenty-four native C++ tests and sixty dashboard tests passing.

The Metatron results are reported as comparative evaluations against a reference tracking pipeline, with higher reach success rates and greater resistance to pushes, and with the same approach producing climbing and acrobatic motions for a simulated character.

Two cautions belong alongside these numbers. First, the release notes state that the anatomical generative prior has no rollout-quality evaluation and that its card declares training simulator only. Second, they state that the native teleoperation, tracking and dashboard were not run on the physical robot in that release. These are the project's own disclosures, and they define exactly how much of the reported capability has been demonstrated on hardware.

7. Role in Physics-AI Workflows

The framework occupies a specific position in a physics AI programme: it is the layer where a physics engine becomes a learning environment, and where a learned policy becomes a deployed controller.

In the training stage, its contribution is making the physics engine interchangeable. A policy that behaves identically in Isaac Gym, Isaac Lab, Newton and MuJoCo is one whose behaviour is a property of the learned dynamics rather than of an engine artefact. For a team that plans to ship, this is the single most valuable check the framework provides, and it is cheaper to run here than to discover the discrepancy in the field.

In the data stage, the contribution is closing the gap between human motion and robot motion. Retargeting the whole public capture dataset onto a target skeleton in one command, then training on it, is what turns a policy from "tracks this one clip" into "has the skill".

In the deployment stage, the contribution is that the observation computation travels with the policy. Baking observation processing into the exported ONNX model removes a whole class of mismatch between training and deployment, and the native runtime adds a safety supervisor and a qualification pipeline on top of that.

The generative-policy direction is the newest and most consequential. The idea of a frozen general motion policy plus programmable skill code changes the economics of humanoid task development: new tasks become code rather than new training runs. The companion paper on Metatron describes exactly this split, with a frozen policy converting partial body goals, velocity commands and style cues into coordinated motion while ordinary stateful code supplies the task intent. The physics AI learning path places this in a broader sequence of skill acquisition, and PyVista covers the visualisation side that makes simulated behaviour inspectable.

For a programme that treats the simulator as part of the physics model, the framework's most valuable contribution is arguably the changelog itself: a maintained, dated record of what each backend actually did, which is the raw material for building a credible sim-to-real argument.

8. Limitations and Failure Modes

The first limitation is that the strongest claims are the least verified. The generative prior shipped in November 2026 has no rollout-quality evaluation, and the native deployment features were validated in simulation rather than on the physical robot. A team planning a hardware pilot should treat the framework as providing the tooling for that validation, not as having completed it.

The second is that the refactor broke compatibility deliberately. The October 2026 restructuring replaced the environment model, the configuration system and the authoring style, and saved configurations from earlier versions must be regenerated. Long-running experiments that relied on saved configurations need to be re-resolved, and the November 2026 release notes itemise which saved configurations changed, which weights changed, and which setups now exhibit numerical differences because of corrected backend behaviour. This is a healthy engineering choice and a disruptive one.

The third is the physics divergence across backends. The list of per-backend corrections is long enough that identical policies produce measurably different rewards, efforts and velocities depending on the engine. For example, the anatomical humanoid's body angular-speed cap is read in degrees per second by one backend and radians per second by another, and is ignored by the remaining two. A cross-engine comparison therefore needs the differences accounted for, or the comparison itself becomes a source of error.

The fourth is asset fidelity. The MuJoCo XML standardisation is a substantial simplification and it works, but the render-versus-physics split means high-fidelity rendering is decorative, and the removal of the Genesis backend removes an option some users had adopted.

The fifth is the experimental status of the native runtime. It is explicitly a prototyping reference, its TensorRT path always runs as captured CUDA graphs, and the release notes list which parts were never exercised on hardware.

The sixth is a general limitation of learned policies that the framework inherits rather than solves: a policy trained in simulation is bounded by the fidelity of the simulator and by the distribution it was trained on. The operating limits and numerical stability collection describes how such policies are validated, and the physics-based deep learning entry covers the modelling assumptions underneath.

9. Version and Source Notes

The release history as published: version 3.3 on 10 November 2026, version 3.2 on 11 August 2026, version 3.1 on 18 February 2026, version 3.0 on 12 January 2026, and version 2.3 on 10 October 2025. An additional release dated 13 August 2026 sits inside the 3.2 line and contains the substantive backend migration.

The 13 August 2026 release switched to Newton 1.0.0 and to Isaac Lab 3.0 with Isaac Sim 6.0, made the MuJoCo XML specification the single source of truth for robot assets, fixed friction randomisation and combine-mode conversion across the Newton, Isaac Gym and Isaac Lab backends, added PyPI and uv installation support, and refreshed the pretrained artefacts with resolved configuration sidecars.

The 10 November 2026 release shipped a generative prior for the anatomical SOMA humanoid, moved the G1 hand frame to the wrist, registered the G1 with Dex3 hands as a robot with 47 bodies and 43 degrees of freedom, extended the native deployment with hand teleoperation and a live metrics dashboard, and corrected a set of simulator backend behaviours found by review. It also retrained the anatomical tracker with explicit head tracking, reducing the reported head pitch error from 18.7 to 2.2 degrees, and raised the anatomical body angular-speed cap from 100 to 1000 to match the other humanoids.

The 28 October 2026 announcements on the main branch were two: the release of Metatron, the generative whole-body policy for the Unitree G1, together with its paper, project page, browser demo and downloadable weights; and a major refactor of the environment into explicit step phases over modular state owners, with a cleaner configuration system, declarative authoring of observations, rewards and terminations as pure tensor kernels, per-channel sensor declarations carrying measurement noise and a state-estimation stage, and the removal of the Genesis backend.

On licensing: the source code is released under the Apache 2.0 licence, while NVIDIA-provided pretrained models are released under the NVIDIA Open Model Licence unless a model card states otherwise. Each model directory includes a model card, licence and attribution notice, and third-party software and bundled asset notices are listed separately, covering the robot hardware, the imitation learning codebase, Isaac Lab and the body model assets.

On sources: version facts and behaviour changes come from the repository's release notes and commits. The architecture, the modular component model, the backend roster and the licensing split are the project's own documentation. The training-time and accuracy figures in Section 6 are the project's own validation results, quoted with their validation procedure and with the project's stated limitations attached.

10. References

  1. ProtoMotions repository README, NVlabs on GitHub, framework overview, generative policies, deployment and rendering features.
  1. ProtoMotions release v3.3 notes, 10 November 2026, anatomical SOMA generative prior, G1 wrist frame, Dex3 robot, backend corrections and validation results.
  1. ProtoMotions release of 13 August 2026, Newton 1.0.0 and Isaac Lab 3.0 migration, MJCF as asset source of truth, installation support.
  1. ProtoMotions main-branch announcements, 28 October 2026, Metatron release and the environment refactor.
  1. METATRON: A Promptable Whole-Body Execution Layer for Humanoids, arXiv 2610.38027, frozen generative policy with programmable skill layer.
  1. ProtoMotions installation guide and quickstart, project documentation, virtual environment setup and backend-specific requirement files.
  1. ProtoMotions G1 deployment tutorial, project documentation, data preparation through real-robot deployment and the native runtime.
  1. ProtoMotions generative pre-trained controller and parameter-efficient fine-tuning guide, project documentation.
  1. ProtoMotions robot configuration guide, project documentation, adding a robot through the MuJoCo XML asset and the configuration class.
  1. NVIDIA Open Model Licence, licensing terms for the pretrained models and their attribution requirements.
Tags: