HALO

New
License: Apple proprietary license (non-OSI; GitHub marks Other/NOASSERTION; see repository LICENSE) · Updated 2026-11-15
My rating

1. Overview

HALO is a perceptive humanoid autonomy system published by Apple's Machine Learning Research group on November 4, 2026, alongside the paper "Perceptive Humanoid Motion from Simulation" (arXiv:2611.06908) and a fully open code release in the apple-aiml-research/ml-halo repository. The system lets a humanoid robot follow natural-language instructions and first-person vision to perform whole-body motions in real environments that were never scanned, mapped, or used in training. Validation ran on an off-the-shelf Unitree G1 body, not an Apple robot, and reported a 91 percent mean single-task success rate across 360 real trials in six unseen homes, with 70 percent on chained multi-step tasks. HALO is notable for being trained entirely on synthetic data: the authors generate roughly 300 hours of paired vision-language-motion data across 1,000 procedurally created homes, so the dataset scales with compute and not with human labor. The release covers the full pipeline of data generation, model training, and deployment code, although no trained tracker weights are shipped and must be reproduced with the provided training recipes. The project page is apple-aiml-research.github.io/ml-halo, and the paper and code together make HALO one of the most complete open releases of a sim-to-real humanoid stack to date.

2. How It Works

HALO decomposes the path from a spoken instruction to a completed motion into three asynchronous layers, each with a distinct responsibility and model. The top layer is a task orchestrator built on a vision-language model. It defaults to Qwen3.6-35B-A3B, served locally in 4-bit MLX format on a Mac mini M4 Pro (64 GB) carried on the robot, so the loop runs without a network connection; for heavier reasoning it can call a cloud API instead. The orchestrator watches the robot's egocentric camera feed, resolves ambiguous instructions into a concrete subtask, checks whether the current action succeeded, and advances or retries the plan, handing one explicit subtask at a time to the layer below. The middle layer is a motion generator that combines a frozen Qwen3.5-2B text encoder with a diffusion (DiT) decoder and, given the current view and the active subtask, plans about five seconds (150 frames) of whole-body motion, re-planning roughly three times per second so the motion continuously corrects against the environment. A "conditioning on executed frames" mechanism keeps the plan continuous across replans. The bottom layer is a 50 Hz motion tracker trained with reinforcement learning; it receives depth images and the robot's own proprioceptive state and outputs 29 joint-position targets to the G1's low-level PD controller, providing terrain awareness and a fall-prevention closed loop. The tracker is trained in three phases in Isaac Lab: privileged experts, multi-teacher distillation, and multi-critic fine-tuning.

3. Scope and Assumptions

The first boundary is what ships. The repository contains code, not trained weights: the tracker students are not released, and reproducers must train them with the in-repo pipeline; the G1 body, cameras, and on-board compute are not part of the code. The orchestrator model is a stock community 4-bit quantization (mlx-community/Qwen3.6-35B-A3B-4bit, about 20 GB) and requires Apple silicon with at least 32 GB memory. The second boundary is the task vocabulary: the demonstrations rest on a fixed set of about ten instruction templates (sit on an armchair, walk to a trash bin, crawl under a table, push a box, and so on), and the orchestrator maps a user's free-form wording onto one of these; adding a new skill still requires a human-authored prompt and action keyframes, so the catalogue grows only as fast as people can write them. The third boundary is sensing: HALO is vision-first and has no tactile feedback, so once an object leaves the camera field of view or is occluded, the policy loses its signal about where that object is, and contact-rich tasks degrade; the authors name tactile sensing as the natural next step. A fourth practical note is licensing: the code is released under Apple's own terms (GitHub marks it Other/NOASSERTION, not an OSI-approved open-source license), a personal non-exclusive license allowing use, modification, and redistribution with conditions, while trademark and patent rights are not granted, and third-party assets such as the G1 description files retain their original licenses.

4. Getting Started

The quickest entry point is the repository README at github.com/apple-aiml-research/ml-halo, which maps each directory to a paper section. The six directories are datagen (Infinigen scenes, Kimodo motion, Isaac and MuJoCo rendering), generator (Qwen3.5-2B plus DiT training and open-loop evaluation), tracker (Isaac Lab three-phase training plus Python and C++ real-time deployment), orchestrator (the halo_orchestrator language planner co-served with the generator), system (closed-loop run_sim.py and run_real.py launchers, halo_comm messaging, halo_serve on-device scheduling, and a live web viewer), and assets (G1 MJCF, URDF, and USD descriptions plus scene objects). A first experiment is to run the closed loop in MuJoCo with run_sim.py, then move to the real G1 with run_real.py after training the tracker. The project page apple-aiml-research.github.io/ml-halo provides the system description, gallery, training overview, and the overview video. Because no tracker weights ship, a reproducer should expect to run the tracker training pipeline (privileged experts, then multi-teacher distillation, then multi-critic fine-tuning) on Isaac Lab before any real-robot deployment, and should budget for the stated GPU footprint: data synthesis on about 250 H100 GPUs for roughly 48 hours, the generator on 64 B200 GPUs, and the tracker on 4 H100 GPUs.

5. Comparison with Alternatives

HALO sits in the "simulation school" of robot learning, alongside platforms such as NVIDIA Isaac Sim and efforts like GR00T, and contrasts with the "real-data school" that collects teleoperation or human-video footage. Its defining architectural choice is modularity: instead of a single end-to-end vision-language-action (VLA) model mapping pixels directly to action, HALO separates understanding (orchestrator VLM), generation (diffusion motion model), and tracking (RL controller), and only passes a subtask description and a roughly five-second motion plan between layers. That separation lets each layer be replaced or trained independently: a stronger orchestrator model can be swapped in without retraining the motion stack, and the tracker can be hardened with reinforcement learning for high-frequency, stable execution. Against end-to-end VLA systems such as Physical Intelligence's pi0.5 or NVIDIA's GR00T, HALO trades some end-to-end generality for interpretability and per-layer trainability, and it makes the "unseen environment" problem tractable because the policy never relied on a pre-scanned map. The paper also documents ablations that expose the load-bearing pieces: removing plan continuity drops sit success from 84 to 24 percent, swapping the DiT decoder for a same-size regression decoder collapses it to 19 percent, and removing depth from the tracker drops it to 74 percent.

6. Benchmarks and Evidence

The headline numbers come from the paper's Table II. On single tasks, HALO reports 91 percent mean success in held-out simulated environments (6 task types across 100 rooms, 10 trials each, 6,000 trials) and 91 percent in real environments (6 task types across 6 unmapped real homes, 10 trials each, 360 trials) - the sim and real figures converge, suggesting the trained capability transfers. On chained tasks, three two- or three-step combinations averaged 70 percent in one real home (10 trials each), lower than the product of single-task rates, indicating that subtasks are not independent and long-horizon hand-offs are the weak point; the Crouch-Go-To-Push chain reached only 50 percent. Training scale, reported in the appendices, is modest by large-model standards: 1,000 Infinigen procedural homes, a 296.3 hour synthetic corpus (162,698 episodes, rounded to about 300 hours in the text), with data synthesis using about 250 NVIDIA H100 GPUs for roughly 48 hours, the generator trained on 64 B200 GPUs, and the tracker on 4 H100 GPUs. These are developer-reported, task-scoped evaluations rather than an independently reproduced leaderboard; the numbers should be reproduced locally before appearing in a procurement document, and the authors are explicit that the single-task figure is a curated mean rather than a guaranteed deployment rate.

7. Role in Physics-AI Workflows

In a Physical AI programme, HALO is a concrete, fully scripted example of the simulation-to-reality pipeline: synthetic data generation, imitation and reinforcement training, and deployment on commodity hardware, with the "brain" (inference) separated from the "body" (actuation). Its explicit claim that the data corpus scales with compute and not with human labor speaks directly to the data-bottleneck debate in robot learning, and the use of an off-the-shelf body (Unitree G1) plus open models (Qwen) plus a consumer mini-PC shows that the autonomy stack can be assembled rather than built from scratch. For teams building Physical-AI products, HALO's value is less a turnkey robot and more a reproducible engineering recipe: a standard interface between a language planner, a motion generator, and a joint-level tracker, with the whole chain open for audit and modification. The site already covers the simulation engines HALO renders against, such as NVIDIA Warp and MuJoCo, the large-scale behaviour-cloning references RoboCaSa and SuperMimic, and the broader platform context appears in the NVIDIA and Physical Intelligence company reports.

8. Limitations and Failure Modes

The most cited failure mode is the absence of tactile sensing: because HALO relies on vision alone, objects leaving the field of view or becoming occluded remove the contact signal, and contact-rich tasks degrade. A second mode is long-horizon degradation, where chaining subtasks multiplies the chance of a single failure (the Crouch-Go-To-Push chain fell to 50 percent), exposing that subtasks are coupled and not independent. A third mode is the narrow inter-layer interface and a relatively simple orchestrator: the system maps free language onto about ten fixed instruction templates, and adding skills still needs human-written prompts and keyframes, so coverage grows slowly. A fourth mode is reproducibility friction: no trained weights ship, the tracker must be self-trained, and the orchestrator depends on an Apple-silicon 4-bit model, so "open code" is not "ready to run"; hardware - the G1 body, cameras, and backpack computer - is outside the repository entirely. A fifth mode is evaluation scope: the 91 percent is a mean over six scripted single tasks in unmapped homes, not a measure of generalisation across the full space of daily activities, and the paper is explicit that the result is research-scoped. As with any research release, HALO's results are paper-specific and not a general deployment guarantee.

9. Version and Source Notes

Record one: HALO was published by Apple's Machine Learning Research group; the paper "Perceptive Humanoid Motion from Simulation" is arXiv:2611.06908 (v1, November 4, 2026), authored by Siyuan Hu, Abulikemu Abuduweili, Yue Meng, Shipeng Liu, Quentin Leboutet, Chen Chen, Matthias Muller, and Vladlen Koltun. Record two: the code repository is github.com/apple-aiml-research/ml-halo, created around November 4-5, 2026, with a single main branch and a small number of commits; the project page is apple-aiml-research.github.io/ml-halo. Record three: the license is Apple's own terms (GitHub marks Other/NOASSERTION, not an OSI license); use, modification, and redistribution are permitted with conditions, trademark and patent rights are not granted, and third-party assets keep their original licenses. Record four: no trained tracker weights ship; the orchestrator model is the community 4-bit quantization mlx-community/Qwen3.6-35B-A3B-4bit and requires Apple silicon with at least 32 GB memory. Record five: the six repository directories - datagen, generator, tracker, orchestrator, system, assets - map one-to-one to the paper's method sections, which makes the codebase readable alongside the text. Record six: the validation body is an unmodified Unitree G1 with an added wide-angle camera (OAK-1W) and a RealSense D435i depth camera, and an on-board Mac mini M4 Pro plus a Jetson Orin NX; these hardware additions are documented but are not included in the code release.

10. References

  1. Hu S., Abuduweili A., Meng Y., Liu S., Leboutet Q., Chen C., Muller M., Koltun V., Perceptive Humanoid Motion from Simulation, arXiv:2611.06908, 2026.
  1. Apple Machine Learning Research, HALO code repository, github.com/apple-aiml-research/ml-halo, README, directory docs, and citation.
  1. Apple Machine Learning Research, HALO project page, apple-aiml-research.github.io/ml-halo, system description, gallery, and overview video.
  1. arXiv, Perceptive Humanoid Motion from Simulation, arxiv.org/abs/2611.06908.
  1. China Galaxy Securities, Physical AI and Robotics sector note: Apple HALO paper, a heavyweight result for the simulation school, 2026-11.
  1. OFweek Intelligent Hardware, Apple open-sources robot "brain" HALO: tested on Unitree G1, 91 percent single-task success, 2026-11.
  1. Robotics Guide, Apple's HALO Teaches a Unitree G1 to Navigate Unseen Rooms With Simulation-Generated Data, 2026-11.
  1. Unitree Robotics, G1 humanoid robot product page, unitree.com, validation hardware platform.
  1. NVIDIA, Isaac Lab training framework, developer.nvidia.com/isaac/isaac-lab, used for the three-phase tracker training.
  1. Infinigen, procedural scene generation, github.com/princeton-vl/infinigen, the scene engine behind the datagen pipeline.
Tags: