Leo Thakur
Back to projects

Self-driving car

I trained a neural network to drive by recording myself driving in the DonkeyCar simulator. It takes a 160×120 camera frame and outputs steering and throttle, 20 times a second. The recordings below show the model in the simulator, along with how it works and its training results.

Watch it drive

One full lap of Mini Monaco in the Donkey simulator, driven by my newest model, mypilot_mix. No one is on the controls.

AUTOPILOT 0.0 s

The camera feed

This is exactly what the model sees: 160×120 pixels. The needle is the steering it chose for each frame.

It jumps between positions because it's a categorical model. It picks one of 15 steering bins instead of any value from -1 to 1.

The lap from above. The dot is the car.
Model steering
+0.00
Direction
Straight

How it learns

This is behavioral cloning. There's no code saying "if the line curves left, turn left." The network gets tens of thousands of camera frames, each paired with what I was doing with the wheel at that moment, and learns to copy me.

Drive laps by hand

I drive the simulator with a PXN V900 racing wheel. DonkeyCar doesn't support that wheel, so I wrote a joystick class that maps its axes and buttons.

mysim/my_joystick.py

class PXNV900Joystick(Joystick):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.axis_names = {
            0x00: 'steering',
            0x02: 'gas',
            0x05: 'brake',
        }
        self.button_names = {
            0x130: 'A',
            0x131: 'B',
            0x136: 'left_paddle',
        }

Training results

I've trained five models on the simulator data. The chart is mypilot_cat2, my best model before the newest one. Training loss is error on the data it learns from. Validation loss is error on data it never sees during training.

mypilot_cat2 loss by epoch

Categorical cross-entropy, lower is better. Hover to see each epoch.

Show the numbers as a table
EpochTrainingValidation

Most of the drop happens in the first 10 epochs. After that both lines flatten and stay close, so it isn't just memorizing. Best validation loss was at epoch 81. Early stopping ended it at 91 after 10 epochs without improvement.

ModelTypeDataEpochsBest val. lossExtras
mypilot (1st)Lineardata/440.0429None
mypilot (2nd)Lineardata/310.0416None
mypilot_catCategoricaldata/610.7724Brightness, blur
mypilot_cat2Categoricaldata_aug/ (flipped)910.8241Brightness, blur
mypilot_mixCategoricaldata/ + simulator laps590.7468Brightness, blur

Linear and categorical models use different loss functions (squared error vs. cross-entropy), so only compare within a type.

Epoch by epoch

An epoch is one pass over all the training data. I trained mypilot_mix a second time and kept a copy of the model after every epoch. Then I let the copies drive Mini Monaco from the same start. Here is one run each from three of them.

EPOCH 112 s

After 1 epoch

It gets through the first corner, then drives into a wall 9 seconds in and stops.

19 m · no lap

EPOCH 322 s

After 3 epochs

It scrapes a wall early and keeps going. It gets about three quarters of the way around before it hits another wall and stops.

54 m · no lap

EPOCH 1427 s

After 14 epochs

A full lap without touching a wall, with the throttle turned down to 75%. This is the lap at the top of the page.

70 m · lap in 25.66 s

Each clip is a single run. The same copy can crash on one try and finish a lap on the next, so I run each one a few times. Most runs still end at a wall.

One model, seven tracks

My latest model, multi_r4, drives seven different tracks in the simulator. To train it, a script that follows a recorded line drove laps on every track and labeled each camera frame with its steering and speed. Then the model drove by itself while the script corrected it, and those corrections went back into training. Leaving the lane, hitting a wall, or getting stuck reset the car.

Each clip is the 160×120 camera the model drives from, next to a map of its route. It picks its own steering and throttle.

Generated Roads

run 83.0 s · 240 m · avg 2.89 m/s

Mini Monaco

lap 32.5 s · 64 m · avg 1.97 m/s

Warehouse

lap 20.0 s · 42 m · avg 2.10 m/s

Mountain

lap 30.9 s · 59 m · avg 1.92 m/s

Robo Racing League

lap 32.0 s · 43 m · avg 1.35 m/s

Generated Track

lap 16.7 s · 25 m · avg 1.47 m/s

Waveshare

lap 11.9 s · 18 m · avg 1.51 m/s