Self-driving car
I trained a neural network to drive by recording myself driving in the DonkeyCar simulator. It takes a 160×120 camera frame and outputs steering and throttle, 20 times a second. The recordings below show the model in the simulator, along with how it works and its training results.
- Demo environmentDonkey simulator
- PlatformDonkeyCar 5.3
- LanguagePython 3.11
- ModelKeras CNN
- Training data56,474 frames
- Autopilot lap25.66 s
Watch it drive
One full lap of Mini Monaco in the Donkey simulator, driven by my newest model, mypilot_mix. No one is on the controls.
The camera feed
This is exactly what the model sees: 160×120 pixels. The needle is the steering it chose for each frame.
It jumps between positions because it's a categorical model. It picks one of 15 steering bins instead of any value from -1 to 1.
- Model steering
- +0.00
- Direction
- Straight
How it learns
This is behavioral cloning. There's no code saying "if the line curves left, turn left." The network gets tens of thousands of camera frames, each paired with what I was doing with the wheel at that moment, and learns to copy me.
Drive laps by hand
I drive the simulator with a PXN V900 racing wheel. DonkeyCar doesn't support that wheel, so I wrote a joystick class that maps its axes and buttons.
mysim/my_joystick.py
class PXNV900Joystick(Joystick):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.axis_names = {
0x00: 'steering',
0x02: 'gas',
0x05: 'brake',
}
self.button_names = {
0x130: 'A',
0x131: 'B',
0x136: 'left_paddle',
}
Record every frame
While I drive, DonkeyCar saves every camera frame with my steering and throttle at that moment. Recording starts whenever the throttle is down. My main dataset is 35,059 frames.
mysim/data/catalog_0.catalog
{"_index": 0,
"_session_id": "26-05-28_0",
"cam/image_array": "0_cam_image_array_.jpg",
"user/angle": 0.0,
"user/mode": "user",
"user/throttle": 0.0}
Mirror the data
If a track has more right turns than left, the model learns to lean right. So I flip every image horizontally and negate its steering angle. That doubles the data and evens out the turns.
mysim/flip_augment.py
""" Double the training dataset by horizontally flipping all images and negating steering angles. Writes to data_aug/ directory. """ from PIL import Image, ImageOps img = ImageOps.mirror(Image.open(src)) new_r["user/angle"] = -r["user/angle"] # negate steering
Train
A convolutional neural network learns to map frames to steering and throttle. Random brightness and blur during training keep it from depending on exact lighting.
mysim/myconfig.py
DEFAULT_MODEL_TYPE = 'categorical' AUGMENTATIONS = ['BRIGHTNESS', 'BLUR'] EARLY_STOP_PATIENCE = 10 BATCH_SIZE = 64 LEARNING_RATE = 0.0005
Autopilot
Load the trained model and switch to full auto. Same drive loop, but steering and throttle now come from the network instead of the wheel.
terminal
python manage.py drive --model models/mypilot_cat2.h5 --type categorical
Training results
I've trained five models on the simulator data. The chart is mypilot_cat2, my best model before the newest one. Training loss is error on the data it learns from. Validation loss is error on data it never sees during training.
mypilot_cat2 loss by epoch
Categorical cross-entropy, lower is better. Hover to see each epoch.
Show the numbers as a table
| Epoch | Training | Validation |
|---|
Most of the drop happens in the first 10 epochs. After that both lines flatten and stay close, so it isn't just memorizing. Best validation loss was at epoch 81. Early stopping ended it at 91 after 10 epochs without improvement.
| Model | Type | Data | Epochs | Best val. loss | Extras |
|---|---|---|---|---|---|
mypilot (1st) | Linear | data/ | 44 | 0.0429 | None |
mypilot (2nd) | Linear | data/ | 31 | 0.0416 | None |
mypilot_cat | Categorical | data/ | 61 | 0.7724 | Brightness, blur |
mypilot_cat2 | Categorical | data_aug/ (flipped) | 91 | 0.8241 | Brightness, blur |
mypilot_mix | Categorical | data/ + simulator laps | 59 | 0.7468 | Brightness, blur |
Linear and categorical models use different loss functions (squared error vs. cross-entropy), so only compare within a type.
Epoch by epoch
An epoch is one pass over all the training data. I trained mypilot_mix a second time and kept a copy of the model after every epoch. Then I let the copies drive Mini Monaco from the same start. Here is one run each from three of them.
After 1 epoch
It gets through the first corner, then drives into a wall 9 seconds in and stops.
19 m · no lap
After 3 epochs
It scrapes a wall early and keeps going. It gets about three quarters of the way around before it hits another wall and stops.
54 m · no lap
After 14 epochs
A full lap without touching a wall, with the throttle turned down to 75%. This is the lap at the top of the page.
70 m · lap in 25.66 s
Each clip is a single run. The same copy can crash on one try and finish a lap on the next, so I run each one a few times. Most runs still end at a wall.
One model, seven tracks
My latest model, multi_r4, drives seven different tracks in the simulator. To train it, a script that follows a recorded line drove laps on every track and labeled each camera frame with its steering and speed. Then the model drove by itself while the script corrected it, and those corrections went back into training. Leaving the lane, hitting a wall, or getting stuck reset the car.
Each clip is the 160×120 camera the model drives from, next to a map of its route. It picks its own steering and throttle.
Generated Roads
run 83.0 s · 240 m · avg 2.89 m/s
Mini Monaco
lap 32.5 s · 64 m · avg 1.97 m/s
Warehouse
lap 20.0 s · 42 m · avg 2.10 m/s
Mountain
lap 30.9 s · 59 m · avg 1.92 m/s
Robo Racing League
lap 32.0 s · 43 m · avg 1.35 m/s
Generated Track
lap 16.7 s · 25 m · avg 1.47 m/s
Waveshare
lap 11.9 s · 18 m · avg 1.51 m/s