How Neural Networks Learn: Backpropagation from One Weight

A neural network learns by repeating five steps. It makes a prediction, measures how wrong it was, works out which way to change each weight to be less wrong, changes them a little, and goes again. That is the whole idea. The rest of this post is the detail of each step, on an example small enough to do by hand.

The plan is to run this loop on one weight first, then on two weights in a chain, then on a real network you can watch learn in the page. At each stage the numbers are computed, not made up, and the demos use the same code as the diagrams.

You need to be comfortable with slopes. If not, the derivative section of the maths post covers all that is used here.

1. A model is a function with knobs

A model takes an input and gives an output. The knobs that decide what it does are called weights. Training means finding good settings for the weights.

The smallest possible model has one weight:

• prediction = w × x

Our example is fixed for the whole post. The input is x = 2 and the answer we want is 10. The weight starts at w = 3. With that weight the model predicts 3 × 2 = 6, which is wrong. The right weight is 5, because 5 × 2 = 10. Training has to find that, using only the size of the mistake.

A real network is the same thing with millions of weights and many inputs. Everything below carries over.

2. Error and loss

Error is how far off the prediction is: prediction − target. Loss is one number that scores the mistake. We use the squared error: loss = error².

How to do it

1. Run the model forward to get a prediction.
2. Subtract the target to get the error.
3. Square the error to get the loss.

Example: w = 3

1. Prediction: 3 × 2 = 6.
2. Error: 6 − 10 = −4.
3. Loss: (−4)² = 16.

Why square it? A prediction that is too high and one that is too low should both count as bad, and squaring removes the sign. It also punishes big mistakes more than small ones, and it gives a smooth curve, which matters in the next step. With many training examples, the loss is the average over all of them.

The usual mistake: mixing up error and loss. The error can be negative. The loss never is, and training tries to push it to zero.

3. The gradient: which way is downhill

Plot the loss against w and you get a valley, with the lowest point at w = 5, where the loss is 0.

The gradient is the slope of that curve at the current weight. It answers two questions at once. The sign says which way the loss goes up, and the size says how steeply.

How to do it

For this model the slope has a formula, from the chain rule: gradient = 2 × error × x. If you want to see where it comes from, the loss is (w × x − 10)². The derivative of a square is 2 times the thing inside, and the thing inside changes by x for each unit of w.

Example: w = 3

• Gradient = 2 × (−4) × 2 = −16.

A negative slope means the loss goes down when w goes up. So w should increase.

Checking it without any formula. Nudge the weight by a tiny amount and watch the loss. At w = 3.001 the prediction is 6.002, the error is −3.998 and the loss is 15.984004. The loss fell by 0.015996 for a nudge of 0.001, a slope of about −15.996. Nudging both sides, by 0.001 either way, gives −16 almost exactly. The formula and the nudge agree. Keep this check in mind: it is how people test that their backpropagation code is right.

Try it. Drag w and watch all four numbers, and the slope, change.

The usual mistake: reading the gradient as the direction to move. It points uphill. You move the opposite way.

4. Gradient descent: step downhill

Gradient descent uses the gradient to change the weight. The rule is one line:

• new w = old w − learning rate × gradient

The learning rate sets the size of the step. The minus sign is the "go the opposite way from uphill" from above.

How to do it

1. Compute the gradient at the current w.
2. Multiply it by the learning rate.
3. Subtract the result from w.
4. Repeat.

Example: learning rate 0.05, starting at w = 3

1. Gradient −16. New w = 3 − 0.05 × (−16) = 3 + 0.8 = 3.8.
2. At w = 3.8 the prediction is 7.6, the error is −2.4 and the loss is 5.76. It was 16.
3. Gradient 2 × (−2.4) × 2 = −9.6. New w = 3.8 + 0.48 = 4.28.

The loss falls each time, and the steps shrink on their own because the slope flattens near the bottom.

Choosing the learning rate. This is the one setting you have to get right. On this problem each step multiplies the distance from the best weight by 1 − 8 × learning rate. That is 0.6 for a rate of 0.05, so the distance shrinks by 40% a step. A rate of 0.01 gives 0.92, which is safe but slow. A rate of 0.25 gives exactly −1, so w bounces between 3 and 7 forever. A rate of 0.3 gives −1.4, so every step lands further away than the last and the loss grows.

Try it. Pick a rate, press Step, and push the rate above 0.25 to watch it break.

The limit of 0.25 belongs to this tiny problem. For a real network there is no formula for it. People find a working rate by trying a few values and watching the loss.

The usual mistake: adding the gradient instead of subtracting it. The loss then climbs instead of falling.

5. Backpropagation: gradients for a chain of weights

One weight is easy because the weight touches the loss directly. Real networks stack layers, so an early weight affects the loss only by way of everything after it. To get its gradient you need the chain rule: multiply the effect of each link along the way.

Take two weights in a chain: x → h = w1 × x → y = w2 × h, with the same loss (y − 10)². Start at w1 = 0.5 and w2 = 3.

How to do it

1. Forward pass. Compute every value from the input to the loss and keep them.
2. Start at the loss. Its gradient with respect to y is 2 × (y − target).
3. Walk backwards. At each link, the gradient of what is behind equals the gradient you already have, times the local slope of that one link.
4. Weights. A weight's gradient is the gradient at its output, times the value that went into it.

Example: w1 = 0.5, w2 = 3, x = 2

Forward:

1. h = 0.5 × 2 = 1.
2. y = 3 × 1 = 3.
3. loss = (3 − 10)² = 49.

Backward:

1. Gradient at y: 2 × (3 − 10) = −14.
2. Gradient of w2: −14 × h = −14 × 1 = −14.
3. Gradient at h: −14 × w2 = −14 × 3 = −42.
4. Gradient of w1: −42 × x = −42 × 2 = −84.

Both weights are too small, and w1 has the bigger gradient because its effect is multiplied through w2.

Check by nudging. Nudge w1 by 0.00001 each way and measure the loss change, and you get −84 to within a billionth. Do the same for w2 and you get −14. Backpropagation gives the same numbers without any nudging.

Update with a learning rate of 0.01. Both new weights use the old gradients:

• w1 = 0.5 − 0.01 × (−84) = 1.34
• w2 = 3 − 0.01 × (−14) = 3.14

Run it again: h = 2.68, y = 3.14 × 2.68 = 8.4152 and loss = 2.5116. The loss fell from 49 to 2.51 in one step.

Step through it yourself. Change the starting weights and watch every number follow.

Why it goes backwards. Each gradient reuses the one after it. The gradient at y is used for w2 and for h, and the gradient at h is used for w1. Computing from the end once gives every gradient in one sweep, at a cost that is a small multiple of one forward pass. The alternative is the nudge method, which needs a forward pass or two for every single weight. With a million weights that is a million passes. That is the reason backpropagation exists.

Real networks add a bend. Stacked multiplications would collapse into one multiplication, so real layers put a non-linear function, such as tanh or ReLU, between them. In the backward pass that adds one more local slope to the chain. For ReLU the slope is 1 when the value is positive, so with the numbers above nothing changes. The network in the next section uses tanh, whose slope is 1 − tanh².

The usual mistake: updating w2 first and then using the new w2 to compute the gradient for w1. Work out all the gradients first from the old weights, then update everything.

Backpropagation and gradient descent are two different things

This is the point people get wrong most often, and it is a common interview question.

• Backpropagation computes the gradients. It is the method for finding how much each weight affects the loss.
• Gradient descent uses the gradients to change the weights. It is the update rule.

You can run gradient descent with gradients from somewhere else, and you can use backpropagation for something other than plain gradient descent, such as an optimiser that adds momentum. In the loop picture, backpropagation is step 4 and gradient descent is step 5.

6. The whole loop on a real network

Now widen the example. The network below has two inputs, a hidden layer of tanh units and one output, and it learns to classify points on a plane. For this job the loss is binary cross-entropy instead of squared error, and the gradient at the output works out to just prediction − label. After that it is the same chain rule you have just done by hand.

Press Play. The shaded map shows what the network currently predicts for every point. Teal means class 1 and red means class 0.

Run these four experiments in order.

1. Two blobs, 0 hidden units. One neuron draws a straight line. It separates the blobs and the loss falls close to zero.
2. XOR, 0 hidden units. XOR puts the two classes on opposite corners, so no straight line can separate them. The loss stays near 0.69, which is what guessing scores, and the accuracy stays near 50%. Marvin Minsky and Seymour Papert pointed out this limit of single-layer perceptrons in their 1969 book Perceptrons.
3. XOR, 4 hidden units. The hidden layer lets the boundary bend. Within a second or two the map turns into four coloured corners and the accuracy reaches 100%.
4. XOR, 2 hidden units. Two units are just enough, but it depends on the random start. Using the same code, learning rate 0.5 and 1,500 steps, 4 hidden units solved XOR from all of the eight random starts I tried, while 2 hidden units failed on two of them. Press New random start to find one.

Also try a learning rate of 2 and one of 0.05, and watch the loss curve change shape. The numbers in the first two experiments were run against the same code that powers the demo.

7. What a framework does for you

Nobody writes the backward pass by hand for a real model. A framework records the forward pass and replays it backwards. Here is the two-weight example in PyTorch:

import torch
 
x, target = 2.0, 10.0
w1 = torch.tensor(0.5, requires_grad=True)
w2 = torch.tensor(3.0, requires_grad=True)
 
h = w1 * x
y = w2 * h
loss = (y - target) ** 2
 
loss.backward()
print(loss.item(), w1.grad.item(), w2.grad.item())

Running it with PyTorch 2.14 on a CPU prints 49.0 -84.0 -14.0: the same loss and the same two gradients as the hand calculation.

The loop for the one-weight example, using PyTorch's built-in gradient descent:

w = torch.tensor(3.0, requires_grad=True)
opt = torch.optim.SGD([w], lr=0.05)
 
for step in range(3):
    loss = (w * 2.0 - 10.0) ** 2
    opt.zero_grad()
    loss.backward()
    opt.step()
    print(step, round(loss.item(), 4), round(w.item(), 4))

It prints 0 16.0 3.8, 1 5.76 4.28 and 2 2.0736 4.568, matching the table in section 4.

The five steps map directly onto the lines:

• Predict and compute the loss: the forward lines.
• loss.backward(): backpropagation.
• opt.step(): gradient descent.
• opt.zero_grad(): clears the gradients, because PyTorch adds new gradients to the old ones. Forgetting this line is a classic bug.

Words you will hear

The demo above uses the whole dataset for every step. Real training uses batches, because the full dataset is too large to process for each update.

Questions you may be asked

• What is backpropagation? An efficient way to compute the gradient of the loss with respect to every weight, by applying the chain rule from the loss back to the inputs and reusing the results.
• What is the difference between backpropagation and gradient descent? Backpropagation computes the gradients. Gradient descent uses them to update the weights.
• What goes wrong if the learning rate is too high or too low? Too high, the loss bounces or grows. Too low, training is correct but takes very long. Section 4 shows both.
• What is the vanishing gradient problem? In a deep network the gradient is a product of many local slopes. If each is small the product shrinks towards zero and early layers stop learning. A sigmoid has a slope of at most 0.25, so a chain of ten of them multiplies the gradient by at most about 0.000001.
• Why can a single neuron not learn XOR? It can only draw a straight line, and no straight line separates XOR. A hidden layer with a non-linear function lets the boundary bend.

The mistakes in one list

• Adding the gradient instead of subtracting it.
• Treating the gradient as the direction to move, when it points uphill.
• A learning rate that is far too high or too low.
• Updating some weights before the other gradients are finished.
• Forgetting to clear gradients in PyTorch.
• Calling backpropagation "the learning". It only computes gradients, and the optimiser does the learning.

Where to go next

• Mini-batches and optimisers such as momentum and Adam, which change step 5 only.
• Loss functions for other jobs: cross-entropy for classes, squared error for numbers.
• Convolutional networks and transformers. They change the shape of the forward pass, and the backward pass is still the same chain rule.

Sources and further reading

• David Rumelhart, Geoffrey Hinton and Ronald Williams, "Learning representations by back-propagating errors", Nature, 1986. The paper that made backpropagation widely known.
• Marvin Minsky and Seymour Papert, Perceptrons, 1969, for the single-layer limit shown in the XOR experiment.
• Andrej Karpathy, micrograd (a small automatic-differentiation engine on GitHub) and his "Neural Networks: Zero to Hero" video series, which builds backpropagation from scratch.
• 3Blue1Brown, the "Neural networks" video series, for gradient descent and backpropagation drawn visually.
• Michael Nielsen, Neural Networks and Deep Learning, a free online book. Chapter 2 explains backpropagation.
• Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016.
• The PyTorch documentation on torch.autograd.