Arca Raven
A simulation-to-real pipeline for autonomous planetary rover navigation — training RL policies in physically accurate environments and deploying them directly onto real hardware. No shortcuts on the physics, no gap between sim and real.
project snapshot
the problem
Autonomous navigation for off-road rovers — planetary or otherwise — is a hard problem with a dangerous failure mode: a policy that looks great in simulation can fail badly on real hardware the first time it encounters real terrain. Most sim-to-real pipelines accept that gap as a fact of life and compensate for it after deployment.
The gap has three specific causes:
Simplified physics
Most simulators abstract away contact forces, suspension dynamics, and soil interaction. Policies trained in simplified physics don't generalise — the real world doesn't abstract anything.
Sensor fidelity mismatch
Sim sensors tend to be clean and noise-free. Real LiDAR, depth cameras, and IMUs aren't. A policy trained on perfect sensor data can't handle real-world sensor noise without explicit simulation of it.
No path from training to deployment
Even when simulation quality is high, there's often no clean export path from trained policy to real hardware runtime. The system and the deployment target are designed separately and bolted together at the end.
the goal
Build a system where the trained policy and the real hardware speak the same language from the start — physically accurate simulation, real sensor noise models, and a structured export path from training checkpoint to Jetson deployment. The loop looks like this:
The benchmark for success isn't a clean training curve — it's a policy that runs identically in simulation and on real rover hardware, with no post-deployment tuning.
what I'm building
Arca RAVEN is structured as two systems that need to be independently excellent and tightly integrated: a simulation and training backend, and a mission control frontend.
Physics Simulation — NVIDIA Isaac Sim
The simulation runs on Isaac Sim with PhysX 5 articulation physics and RTX ray-traced sensor simulation. Rover suspensions are modelled accurately — the rocker-bogie geometry, pivot points, and passive differential are all physically correct. The same physics engine that JPL and ESA use for rover R&D.
Terrain Environments
Terrain stages are generated from real DEM (Digital Elevation Model) data — Mars surface topology sourced from HiRISE and MOLA. Procedurally placed rocks are rigid bodies. Soil is modelled using Bekker's terramechanics SCM model for physically accurate regolith interaction.
Multi-Modal Policy Architecture
The navigation policy processes three sensor streams simultaneously: a 360° LiDAR point cloud through a PointNet++ encoder, a 128×128 depth camera frame through a Conv2D encoder, and an IMU/pose/goal state vector through an MLP. The three representations are fused through a shared network into a continuous velocity command.
RL Training — SAC + Curriculum
Training uses a two-phase curriculum: PPO bootstraps a stable base policy on flat terrain first, then SAC takes over for continuous control on increasingly complex environments. Isaac Lab manages parallel GPU environments (64–256 concurrent) and domain randomisation. TensorBoard streams all reward components live.
Mission Control UI — Tauri + React + Three.js
A Tauri desktop app acts as the command interface. React panels handle scenario selection, training controls, and sensor telemetry. A Three.js overlay reconstructs the simulation state from WebSocket packets at 30 Hz — rover pose, terrain mesh, LiDAR point cloud, and IMU values. The Rust Tauri core manages the Python sim subprocess and all IPC.
Deployment Pipeline — ONNX → TensorRT → ROS2
Trained policies export to ONNX, then optimise to TensorRT for Jetson AGX Orin (INT8 or FP16). The ROS2 inference node subscribes to real sensor topics using the same message types as the simulator — so the node that runs in sim is the same one that runs on hardware.
technical architecture
The system is split across three layers that each have a clear ownership boundary. No layer does the job of another.
Simulation Backend — Python + Isaac Sim
Isaac Sim owns the Python interpreter. A FastAPI server runs in-process on port 8765 and manages sim lifecycle, training sessions, scenario configuration, and WebSocket streaming. The RL environment subclasses Isaac Lab's ManagerBasedRLEnv — curriculum, domain randomisation, reward logging, and checkpointing are all handled there.
Frontend — Tauri + React + Three.js
The Tauri app does not run physics — it is a control and observation panel. The Rust core manages the Python subprocess, IPC commands, and file operations. React + TypeScript handles UI state via Zustand. Three.js renders a live 3D secondary view from WebSocket state packets. Recharts streams training curves in real time.
Hardware Runtime — ROS2 + Jetson
A ROS2 Humble node wraps the exported policy. It subscribes to the same sensor_msgs topics the simulator publishes, runs TensorRT inference, and publishes velocity commands to the motor controller. The node is designed to be topic-identical in sim and on real hardware — no conditional paths, no hardware-specific branches.
engineering decisions
PointNet++ for LiDAR encoding
LiDAR returns a variable-density point cloud — not a fixed-size tensor that a standard CNN can consume directly. PointNet++ handles permutation invariance and partial scans naturally, and its hierarchical local feature extraction is well-suited to terrain classification. The encoder is pre-trained on Mars HiRISE terrain data before RL fine-tuning, which dramatically reduces the policy's sample requirements.
SAC for continuous control — not PPO
Wheel torque and velocity outputs are continuous, and each physics step is computationally expensive — making sample efficiency critical. SAC's off-policy architecture reuses transitions far more effectively than on-policy PPO, and its automatic entropy tuning prevents premature convergence to paths that happen to work on training terrain but don't generalise. PPO is used in stage one of the curriculum because convergence speed on flat terrain matters more than sample efficiency there.
Topic-identical sim and hardware runtime
The most important architectural decision in the whole pipeline was making the ROS2 inference node topic-identical between simulation and hardware. Isaac Sim publishes to the same sensor_msgs types as the real sensor drivers. This means the policy node never needs to know which environment it's running in — and the sim-to-real gap is closed at the interface level, not patched at deployment.
Isaac Sim as the authoritative physics source
The Three.js viewer in the Tauri app renders a reconstruction of the simulation state, not the simulation itself. Isaac Sim's native RTX window remains the authoritative high-fidelity view. This keeps the Tauri app lightweight and avoids the latency and fidelity compromise of running physics in a web renderer — the frontend's job is observation and control, not computation.
Curriculum before complexity
Training directly on complex terrain wastes enormous compute on a policy that can't even navigate flat ground yet. The two-phase curriculum — flat terrain PPO bootstrap, then SAC on progressively rougher environments — gives the policy a stable foundation before introducing the hard problems. Domain randomisation (friction, mass, sensor noise) is layered in during SAC training to harden the policy against real-world variance.
active challenges
This project is currently in the simulation and training phase. These are the problems being worked through right now — not retrospective lessons, but open engineering questions.
VRAM budget during training
Isaac Sim's RTX renderer and 64–256 parallel RL environments compete for the same GPU memory. The target training machine (RTX 3060, 12 GB) requires careful budgeting: dropping to the RayTracedLighting renderer and 128×128 sensor resolution during training recovers enough VRAM to run meaningful parallelism without sacrificing physics accuracy.
Rocker-bogie USD articulation
Getting the articulation tree right for the 6-wheel rocker-bogie is non-trivial. The passive differential bar that links the two rocker arms needs to be modelled correctly in PhysX — it's a passive kinematic constraint, not an actuated joint, and the solver needs to handle it without introducing instabilities in the suspension.
Reward shaping for conservative paths
The reward function needs to balance goal progress against terrain safety — a rover that charges straight at a target and tips over is technically making progress until it isn't. Shaping the reward to prefer stable, energy-efficient paths without making it so conservative that the rover avoids all non-trivial terrain is an ongoing calibration problem.