The Backward Pass¶

The Problem¶

After the forward pass, we have:

• A final output: the loss (a single number measuring how wrong the model was)
• A computation graph: every operation that led to the loss, recorded as Value nodes

Now we need to answer: for every parameter in the model, how much did it contribute to the loss? That is, we need \(\frac{d(\text{loss})}{d(\text{parameter})}\) for every parameter.

The Backward Pass Algorithm (Lines 59–72)¶

def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    self.grad = 1
    for v in reversed(topo):
        for child, local_grad in zip(v._children, v._local_grads):
            child.grad += local_grad * v.grad

This is only 14 lines of code, but it's doing something profound. Let's decompose it.

Step 1: Topological Sort (Lines 60–68)¶

topo = []
visited = set()
def build_topo(v):
    if v not in visited:
        visited.add(v)
        for child in v._children:
            build_topo(child)
        topo.append(v)
build_topo(self)

Why do we need this?¶

Because the backward pass processes nodes from the output back to the inputs. By reversing the topological order (reversed(topo)), we guarantee that every node's gradient is fully computed before we try to propagate it to its children.

Step 2: Seed the Gradient (Line 69)¶

self.grad = 1

self is the loss node — the root of the graph. We set its gradient to 1 because:

The derivative of anything with respect to itself is always 1. This is our starting point.

Step 3: Propagate Gradients Backward (Lines 70–72)¶

for v in reversed(topo):
    for child, local_grad in zip(v._children, v._local_grads):
        child.grad += local_grad * v.grad

For each node \(v\) (starting from the loss, going backwards):

This is the chain rule in action: the gradient of a child = (local gradient) × (parent's gradient). We use += because a node might have multiple parents.

Full Walkthrough¶

Let's trace the backward pass for \(f = \text{relu}(a \times b + d)\):

Result:

The "+=" is Crucial¶

a = Value(3.0)
b = a + a       # a is used twice!

What This Means for Training¶

After calling loss.backward():

• Every parameter (Value in the model) has its .grad set
• This gradient tells us: "nudge this parameter in the opposite direction of .grad to reduce the loss"
• The optimizer then uses these gradients to update all parameters