Linear Layers¶

The Problem¶

We have a 16-dimensional embedding vector x. Now we need to transform it — mix the information in its 16 dimensions to produce a new vector. This is the most fundamental computation in neural networks.

What Is a Linear Layer?¶

A linear layer multiplies an input vector by a weight matrix. It's the neural network's way of "mixing and recombining" information.

Concrete Example¶

A tiny 3→2 example (3 inputs, 2 outputs):

Each output element is a weighted sum of all input elements. The weights determine "how much of each input goes into each output."

The Code (Lines 94–95)¶

def linear(x, w):
    return [sum(wi * xi for wi, xi in zip(wo, x)) for wo in w]

# For each row 'wo' in weight matrix 'w':
for wo in w:
    # Compute the dot product of that row with input 'x':
    sum(wi * xi for wi, xi in zip(wo, x))

Row 0: wo = [0.5, -0.3, 0.1]
       sum(0.5×2, -0.3×3, 0.1×1) = 0.2

Row 1: wo = [0.2, 0.4, -0.2]
       sum(0.2×2, 0.4×3, -0.2×1) = 1.4

Result: [0.2, 1.4]

What's a Dot Product?¶

The inner sum is called a dot product:

It's a measure of "similarity" between two vectors. High dot product = vectors point in the same direction.

Visual Representation¶

flowchart LR
    X["x<br>[x₀, x₁, x₂]"] --> ROW0["row 0: [w₀₀, w₀₁, w₀₂]<br>→ dot product → y₀"]
    X --> ROW1["row 1: [w₁₀, w₁₁, w₁₂]<br>→ dot product → y₁"]

Each row of the weight matrix produces one output element. The number of rows determines the output size.

Where Linear Layers Appear in microgpt.py¶

The linear function is the workhorse — used 7 times per layer, plus one more for the output.

Why "Linear"?¶

Because \(y = Wx\) is a linear function — if you double the input, you double the output. No curves, no bends. Just scaling and mixing.