In progress
Self-driving AI car
Leo taught a neural network to drive by letting it watch him. It gets a 160×120 camera frame, decides how to steer, and does that 20 times a second. This page covers how it works, how training went, what happened on the track, and how to build your own.
- PlatformDonkeyCar 5.3
- LanguagePython 3.11
- ModelKeras CNN
- Training data35,059 frames
- Best lap23.06 s, no driver
Watch it drive
This is one full, uninterrupted lap of the Mini Monaco track in the Donkey simulator. Nobody is touching the controls. Every steering decision comes from Leo's model, mypilot_cat2.
What you're seeing
The video is the model's actual input: 160×120 pixels, recorded as it drove. The needle shows the steering it chose for each frame.
It jumps between fixed positions because this is a categorical model. Instead of outputting any number between -1 and 1, it picks one of 15 steering bins, like a multiple-choice answer.
- Model steering
- +0.00
- Direction
- Straight
How it learns
This is called behavioral cloning. There are no rules like "if the line curves left, turn left." The network sees thousands of examples of a camera frame paired with what Leo did at that moment, and learns to copy him. The whole loop has five steps.
Drive laps by hand
Leo drives the simulator himself using a PXN V900 racing wheel. DonkeyCar didn't support that wheel out of the box, so he wrote a custom joystick class that maps its axes and buttons to steering, throttle, and controls.
mysim/my_joystick.py
class PXNV900Joystick(Joystick):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.axis_names = {
0x00: 'steering',
0x02: 'gas',
0x05: 'brake',
}
self.button_names = {
0x130: 'A',
0x131: 'B',
0x136: 'left_paddle',
}
Record every frame
While he drives, DonkeyCar saves each camera frame with the exact steering and throttle he was using. Recording starts automatically whenever the throttle is pressed. His main dataset has 35,059 of these records.
mysim/data/catalog_0.catalog
{"_index": 0,
"_session_id": "26-05-28_0",
"cam/image_array": "0_cam_image_array_.jpg",
"user/angle": 0.0,
"user/mode": "user",
"user/throttle": 0.0}
Double the data with a mirror
A track that turns right more than left teaches the car to favor right turns. Leo's fix: flip every image left to right and negate its steering angle. Every lap gains a mirror-image twin, without driving another lap.
mysim/flip_augment.py
""" Double the training dataset by horizontally flipping all images and negating steering angles. Writes to data_aug/ directory. """ from PIL import Image, ImageOps img = ImageOps.mirror(Image.open(src)) new_r["user/angle"] = -r["user/angle"] # negate steering
Train the neural network
A convolutional neural network learns to turn a frame into a steering angle and throttle. Training also applies random brightness and blur, so the model doesn't depend on perfect lighting.
mysim/myconfig.py
DEFAULT_MODEL_TYPE = 'categorical' AUGMENTATIONS = ['BRIGHTNESS', 'BLUR'] EARLY_STOP_PATIENCE = 10 BATCH_SIZE = 64 LEARNING_RATE = 0.0005
Hand over the wheel
Leo loads the trained model and switches the car to full auto. The drive loop stays exactly the same. The only change is that steering and throttle now come from the neural network instead of his racing wheel.
terminal
python manage.py drive --model models/mypilot_cat2.h5 --type categorical
Training results
Leo trained four models on the simulator data. The chart shows mypilot_cat2, the one driving the lap above. Training loss measures mistakes on data the model learns from. Validation loss measures mistakes on data it never trains on, which is the honest test.
mypilot_cat2 loss by epoch
Categorical cross-entropy, lower is better. Hover to see each epoch.
Show the numbers as a table
| Epoch | Training | Validation |
|---|
Most of the learning happens in the first ten epochs. After that, both lines flatten out and stay close together. That closeness is a good sign: the model isn't just memorizing its training data. The best validation score came at epoch 81, and early stopping ended training at 91 when ten more epochs brought no improvement.
| Model | Type | Data | Epochs | Best val. loss | Extras |
|---|---|---|---|---|---|
mypilot (1st) | Linear | data/ | 44 | 0.0429 | None |
mypilot (2nd) | Linear | data/ | 31 | 0.0416 | None |
mypilot_cat | Categorical | data/ | 61 | 0.7724 | Brightness, blur |
mypilot_cat2 | Categorical | data_aug/ (flipped) | 91 | 0.8241 | Brightness, blur |
Linear and categorical models measure loss differently (squared error versus cross-entropy), so their numbers can't be compared with each other. mypilot_cat2 scores a bit higher than mypilot_cat because its validation set includes the harder mirrored images.
What happened on the track
Loss numbers only say so much. The real test is letting the model drive. In test runs for this page, here's what happened:
- On Mini Monaco, the track it trained on,
mypilot_cat2completed a clean lap in 23.06 seconds with no crashes. That's the video above. - On other attempts, it tended to clip the wall on one right-hand corner, turning a little too late. That corner is the next thing to fix, most likely with more training laps through it.
- On a different track it had never seen, it left the road within about 30 seconds.
That last result is the most important lesson in this project. A behavioral-cloning model only knows the kind of road it was shown. To drive anywhere, it needs training data from many different tracks, lighting conditions, and surfaces.
Build your own
This guide follows the same path Leo took: start in the free simulator, where crashes cost nothing, then move to a real car. You'll need a computer running Linux, macOS, or Windows and some comfort with the terminal. A game controller helps but isn't required. Check off steps as you go. Your progress is saved in this browser.
-
1
Set up Python
Install Miniconda, then make a separate environment so DonkeyCar's packages don't clash with anything else. Leo uses Python 3.11.
conda create -n donkey python=3.11 conda activate donkey pip install "donkeycar[pc]"
Check that it worked by running
donkey --help. You should see a list of commands. -
2
Install the simulator
Download the simulator for your system from the gym-donkeycar releases page and unzip it somewhere you'll remember. Then install the Python package that lets DonkeyCar talk to it:
git clone https://github.com/tawnkramer/gym-donkeycar pip install -e gym-donkeycar
On Linux, make the simulator executable first:
chmod +x donkey_sim.x86_64. -
3
Create your car
DonkeyCar sets up a project folder with everything a car needs: the drive loop, a config file, and folders for data and models.
donkey createcar --path ~/mysim cd ~/mysim
-
4
Point it at the simulator
Open
myconfig.pyand add these lines. Every setting inmyconfig.pyoverrides the default inconfig.py, so this is the only file you edit.mysim/myconfig.py
DONKEY_GYM = True DONKEY_SIM_PATH = "/path/to/DonkeySimLinux/donkey_sim.x86_64" DONKEY_GYM_ENV_NAME = "donkey-minimonaco-track-v0" AUTO_RECORD_ON_THROTTLE = True
Other tracks include
donkey-generated-track-v0anddonkey-warehouse-v0. Pick one and stick with it while you learn. -
5
Connect a controller
You can drive from the web page DonkeyCar serves at
http://localhost:8887, but a real controller gives much smoother data. Smooth data trains a smoother driver. For common gamepads, set the type inmyconfig.py:USE_JOYSTICK_AS_DEFAULT = True CONTROLLER_TYPE = "ps4" # or "xbox", "F710", ...
For anything else, like Leo's racing wheel, run
donkey createjs. It walks you through pressing each button and moving each axis, then writes amy_joystick.pyyou can edit. SetCONTROLLER_TYPE = "custom"to use it. -
6
Drive and record
python manage.py drive
The simulator opens and you drive. Every frame is saved to
data/while the throttle is pressed. Some tips from Leo's experience:- Drive at least 10 clean laps. More good data beats a fancier model.
- Drive the way you want the car to drive: smooth, centered, consistent speed.
- Include a few gentle recoveries: drift toward the edge, then steer back. Otherwise the model never learns what to do when it's off-center.
- If you crash, use
donkey uito find and delete that stretch before training.
-
7
Mirror your data (optional)
Most tracks turn one way more than the other. Mirroring each frame and flipping the sign of its steering balances that out and doubles your dataset. This is the core of Leo's
flip_augment.py, which writes a new data folder with both versions:from PIL import Image, ImageOps for record in records: img = Image.open(f"data/images/{record['cam/image_array']}") ImageOps.mirror(img).save(f"data_aug/images/flip_{record['cam/image_array']}") flipped = dict(record) flipped["cam/image_array"] = f"flip_{record['cam/image_array']}" flipped["user/angle"] = -record["user/angle"] # negate steering new_records.append(flipped) -
8
Train a model
donkey train --tub ./data --model ./models/mypilot.h5 --type categorical
Training takes minutes to an hour depending on your computer. Keras prints the loss after every epoch, and when it finishes, DonkeyCar saves a plot of the loss next to the model file. Two things to watch:
- Validation loss should fall and then level off. If it starts rising while training loss keeps falling, the model is memorizing. Early stopping handles this for you.
- Try both model types.
linearoutputs smooth numbers, andcategoricalpicks from bins. Leo got his best lap from a categorical model.
-
9
Let it drive
python manage.py drive --model ./models/mypilot.h5 --type categorical
Open
http://localhost:8887and switch the mode to full auto. The model now drives. It will probably make mistakes at first. Note where it fails, drive extra laps through those spots, and train again. That loop is the whole job. -
10
Move to a real car
A physical DonkeyCar is a small RC car with a single-board computer, like a Raspberry Pi, and a camera on top. The official DonkeyCar docs list supported parts and walk through assembly and calibration. The software steps above stay almost the same.
Build in a kill switch before the first run. Make sure you can take back manual control instantly, and start slow. A real car at full throttle can hurt someone or break itself.