Characters as Numbers¶

The Problem¶

We have a list of names like ["emma", "olivia", ...]. But computers can only do math on numbers. We need a way to convert characters to numbers and back.

This conversion process is called tokenization.

What Is a Token?¶

Using characters keeps things simple: there are only ~27 unique characters in our names dataset.

The Code (Lines 23–27)¶

uchars = sorted(set(''.join(docs)))  # unique characters in the dataset
BOS = len(uchars)                     # token id for Beginning of Sequence
vocab_size = len(uchars) + 1          # total unique tokens (+1 for BOS)
print(f"vocab size: {vocab_size}")

Line 24: Building the vocabulary¶

uchars = sorted(set(''.join(docs)))

Let's trace this step by step:

''.join(docs)  # → "emmaoliviaava..." (all names glued together)

set(...)  # → {'e', 'm', 'a', 'o', 'l', 'i', 'v', ...}  (unique chars only)

sorted(...)  # → ['a', 'b', 'c', 'd', ..., 'z']  (sorted alphabetically)

The result is a sorted list of every unique character:

uchars = ['a', 'b', 'c', 'd', 'e', ..., 'y', 'z']
# index:   0    1    2    3    4   ...   24   25

The index of each character in this list becomes its token ID:

• 'a' → 0
• 'b' → 1
• 'e' → 4
• 'z' → 25

Encoding and Decoding¶

flowchart LR
    A["'e'"] -- "uchars.index('e')" --> B["4"]
    B -- "uchars[4]" --> A

The name "emma" becomes:

And \([4, 12, 12, 0]\) can be decoded back to "emma".

Why Sorted?¶

Sorting (sorted(...)) isn't strictly necessary — any consistent mapping would work. But sorting makes the mapping deterministic and predictable, which helps with debugging.

Why Not Just Use ASCII?¶