Reinforcement Learning for Robotics Part 1: CAD to MuJoCo Simulator

420

2026-07-30 | By ShawnHymel

Robot Kits Microcontrollers Displays LCD / TFT M5Stack

Balancing bots have been a solved problem for years thanks to techniques like PID loops. But what if the robot learned to balance on its own, without any hand-tuned control equations? That's exactly what this series is about.

Welcome to Reinforcement Learning for Robotics, a hands-on guide to training a real robot using modern RL techniques, from simulation all the way through to hardware deployment. By the end of the series, you'll have a working remote-controlled balance bot that learned its own control policy entirely through trial and error in simulation. More importantly, you'll have the skills to take these techniques further, including quadrupedal walkers, bipedal robots, or whatever challenge you want to tackle next.

In this first episode, we'll cover what the series will build, why reinforcement learning is worth learning for robotics, what simulation platform we're using and why, and how to get a 3D model of your robot into the MuJoCo simulator, complete with accurate physics. The full project files are available at https://github.com/ShawnHymel/reinforcement-learning-for-robotics.

Why Reinforcement Learning

Traditional robot control methods like PID controllers and Model Predictive Control (MPC) are powerful, but they rely on accurate models and careful hand-tuning for every new situation. Reinforcement learning (RL) takes a different approach: rather than encoding a control policy by hand, the robot discovers one through interaction with a simulated environment, gradually learning behaviors that can be surprisingly robust to conditions it has never seen before.

This is the same class of techniques behind Boston Dynamics integrating RL into Spot's training and Disney's BDX droids learning to walk and express emotions in simulation. We'll go deeper on the theory and mechanics of RL in the next episode.

Prerequisites

Reinforcement learning sits at the intersection of control theory, probability, and deep learning, so there's a non-trivial set of prerequisites. Here's what you'll want to be comfortable with before diving in:

Probability and statistics: Khan Academy has a good course if you need one.
Neural networks: The first three courses of the deep learning specialization on Coursera will give you the foundation you need.
PyTorch: We'll use it throughout the series. Deep Learning with PyTorch by Stevens, Antiga, and Viehmann has excellent worked examples.
Reinforcement learning fundamentals: I recommend looking into the following resources to dive into

Check out the University of Alberta's Fundamentals of Reinforcement Learning course on Coursera,
The above course is based on the classic Sutton and Barto textbook (which is free to download)
My Intro to RL video and blog series also offer good (if brief) starting points

Arduino / ESP32: We'll deploy to an ESP32-based robot. Becky Stern's getting started guide on Digi-Key's site is a great reference.
CAD: We'll use FreeCAD to model the robot. My Intro to FreeCAD series covers the basics if you want to follow along, but you’re not required to use FreeCAD.

Required Hardware

For this series, you’ll need the M5Stack BALA2 Fire self-balancing robot kit.

Choosing a Physics Simulator

Here’s a quick rundown of the current popular simulators you might want to consider:

Gazebo is the long-standing workhorse of the ROS ecosystem. It’s mature and widely supported, but doesn’t handle massively parallelized RL training well.

PyBullet is lightweight and Python-friendly, which made it popular for RL research a few years ago. Development has slowed, and GPU parallelization support is limited.
MuJoCo (Multi-Joint Dynamics with Contact) is the simulator we’ll use throughout this series. It’s the go-to tool for RL researchers, well-documented, and integrates cleanly with Python. The main limitation is that getting GPU parallelization working can be a bit tricky.
Unity ML-Agents had a strong run as a high-fidelity simulation environment with GPU support, but it’s closed-source and has lost ground to newer alternatives.
NVIDIA Isaac Sim is probably the most powerful option available, with first-class GPU acceleration and an industry-grade toolchain. The tradeoffs: closed-source, complex setup, and requires an NVIDIA GPU.
Genesis is new and worth keeping an eye on. It offers AI-assisted scene generation and is developing quickly.

There are plenty more simulators not covered here, and the landscape is changing constantly. For this series, MuJoCo hits the right balance of usability, documentation, and community support. We will use it in CPU mode only, which is easier to get started with and offers maximum compatibility.

Measuring the Robot

Image of Reinforcement Learning for Robotics Part 1: CAD to MuJoCo Simulator

Before we can simulate anything, we need accurate physical parameters for our robot. MuJoCo’s physics engine needs to know the geometry, mass, center of mass, and moments of inertia for each rigid body. That means breaking out a ruler, some calipers, and a small scale.

For the BALA 2 Fire, we measure:

The width, depth, and height of the chassis
The diameter and width of the wheels
The position of the axles relative to the chassis
The mass of the chassis and each wheel (in grams, converted to kilograms for MuJoCo)
The center of mass for each body

For the wheels, we can safely assume uniform density and treat the center of mass as the geometric center. The chassis is more complex. It contains motors, batteries, and electronics, all of which are denser than the surrounding plastic. We estimate the center of mass experimentally by balancing the chassis on a thin edge along each axis and measuring from the axle origin to the balance point.

We also measure the location of the Inertial Measurement Unit (IMU) inside the chassis. It sits beneath the LCD screen and is offset from the center. That offset matters, as it determines how the sensor reads tilt, which directly affects the robot’s ability to learn to balance.

Exporting from FreeCAD

Image of Reinforcement Learning for Robotics Part 1: CAD to MuJoCo Simulator

With measurements in hand, we build a simplified 3D model of the robot in FreeCAD. The model has three bodies: the chassis, the left wheel, and the right wheel. We intentionally keep it simple: a more detailed model increases simulation time without proportionally improving training quality.

One important note: the wheels on the physical robot have ridged treads, but MuJoCo uses convex shapes for contact physics. Modeling those ridges caused the simulator to behave erratically. Instead, we model smooth wheels and tune the friction parameter to approximate the grip of the real tires. This is one of many sim-to-real gaps you’ll encounter, which are places where the simulation and reality diverge, and you have to decide how to bridge them (something we’ll cover in later episodes).

The full code is available in the repository. Within the workspace/mechanical/freecad/scripts/ directory, you’ll find two key scripts:

mesh_export.py: exports each rigid body as an STL mesh file, applying a 90° rotation to convert from FreeCAD’s Y-forward convention to MuJoCo’s X-forward convention
inertia_utils.py: calculates moments of inertia for each body, accounting for non-uniform density and the same axis rotation

To use them, open the FreeCAD Python console (View > Panels > Python Console) and run the following, adjusting paths for your machine:

Copy Code

>>> from pathlib import Path
>>> SCRIPTS_PATH = Path('D:/Projects/GitHub/reinforcement-learning-for-robotics/workspace/mechanical/FreeCAD/scripts')
>>> import sys
>>> sys.path.insert(0, str(SCRIPTS_PATH))

This produces chassis.stl, wheel_left.stl, and wheel_right.stl.

Next, get the inertia data:

Copy Code

>>> import mesh_export
>>> MESHES_PATH = Path("D:/Projects/GitHub/reinforcement-learning-for-robotics/workspace/mechanical/FreeCAD/bala2-fire/meshes")
>>> RPY = (0,0,90)
>>> mesh_export.export_bodies(labels=['chassis', 'wheel_left', 'wheel_right'], output_dir=MESHES_PATH, rpy_degrees=RPY)

Copy the output directly into your MJCF file in the next step.

Building the MJCF File

Most simulators use some kind of descriptor file, often in XML format, to bring everything together: meshes, joints, actuators, etc. The simulator uses this file to know how to load the meshes, how to connect and move them, etc.

The Unified Robot Description Format (URDF) is the standard robot description format supported by simulators like Gazebo, PyBullet, and Isaac Sim. The MuJoCo XML format (MJCF) is unique to MuJoCo, but it accomplishes a similar goal. The main difference is that MJCF keeps more configuration in the file itself. Sensors, actuators, timestep, and contact parameters are all defined in the file rather than inside the simulator. That makes it a bit more verbose, but also more explicit, which is helpful when these parameters matter as much as they do for RL training.

You can read more about the MJCF file format and associated tags here: https://mujoco.readthedocs.io/en/stable/modeling.html.

The MJCF file for this series is in the repository folder workspace/mechanical/freecad/bala2fire/bala2-fire-simplified.xml.

Here are the key concepts:

Timestep: Set to 5 milliseconds. This must match the timestep used during RL training and the control loop frequency on the actual hardware. If inference on the microcontroller is too slow to hit 5ms, you’ll need to increase this and retrain.
Bodies and joints: The chassis is connected to the world via a free joint, allowing unconstrained translation and rotation. The wheels are connected to the chassis via hinge joints, constraining them to rotate around a single axis (the axle).
Actuators: The motors are specified with a normalized control range of -1 to 1, scaled by an estimated gear ratio to arrive at an approximate torque in Newton-meters (~0.05 Nm for the N20 motors with a 1:30 gearbox).
Sensors: MuJoCo has built-in sensor types for accelerometers and gyroscopes. We place both at the IMU site we defined on the chassis body, at the measured offset from center. We also add joint position and velocity sensors to the wheel joints to act as wheel encoders.
Sim-to-real considerations: Friction is estimated at 0.8 for rubber on a hard floor. In a later episode, we’ll use domain randomization to vary this (and other parameters) during training so the policy learns to be robust to uncertainty.

Testing the Model in MuJoCo

In this series, I’ll use Docker to help freeze the versions and guarantee that everything works across all major operating systems. You’re welcome to install everything locally, but I recommend using the Docker image if you want to ensure that everything works just like in the videos.

With the Docker environment running (see the repository README for setup instructions), navigate to /workspace/software/01-test-model-in-mujoco/ and open test_motion.ipynb in JupyterLab via localhost:3000.

Run all the cells in the notebook by pressing shift+enter.

The demo wires up arrow keys to motor speed commands. Use the up/down arrows to try to keep the robot upright. Spoiler: it's extremely difficult to do by hand. It should help give you an intuition for why training an RL agent to handle this automatically is worth the effort.

Image of Reinforcement Learning for Robotics Part 1: CAD to MuJoCo Simulator

The viewer supports mouse controls for zooming and rotating the camera. Press ‘w’ to toggle wireframe mode, which lets you see the IMU site (shown as a small blue box) and verify it's positioned correctly relative to the chassis. Press Backspace to reset the simulation.

Challenge

Before the next episode, try writing a PID controller to balance the robot in simulation. It doesn't need to hold position (it’s OK if it drifts), but it should stay upright. A balance bot is essentially an inverted pendulum on wheels, which is a classic PID problem.

See my two videos if you need a refresher on PID loops:

What’s Next

In the next episode, we'll set up a custom version of CleanRL to train a reinforcement learning agent to balance the robot from scratch using the proximal policy optimization (PPO) algorithm. We'll also introduce curriculum learning, breaking training into phases so the agent can build skills incrementally.