How Claude, ChatGPT, and Gemini Actually Work: Inside a Large Language Model (Part 2)

Building Your Own Foundation Model - Part 2

Part 2 — Why AI Sees Everything as Numbers

"Computers don't understand words. They understand mathematics."

Imagine This...

Suppose you ask Claude:

Explain Artificial Intelligence.

To you, this is a sentence.

To Claude...

It is not English.

It is not grammar.

It is not meaning.

It is simply a sequence of numbers.

Understanding this idea changes how you think about every modern AI model.

Everything Starts With Numbers

Computers cannot process words directly.

They only understand binary.

01101010
10110101
00101101

Everything eventually becomes numbers.

Images.

Audio.

Video.

Text.

Even your question.

From Text to Numbers

Suppose our input is

The cat sat on the mat.

The tokenizer converts it into

Notice something interesting.

The AI never sees the word cat.

It only sees

Interactive Playground 💡

Try typing different sentences into the Tokenization Playground from Part 1.

Observe how changing even one word produces a different sequence of token IDs.

But Numbers Alone Are Useless

Suppose we feed the model

464
3797
3332

What does

mean?

Nothing.

It is just an identifier.

Think about your Aadhaar number.

548273918273

That number doesn't describe you.

It merely points to your record.

Token IDs work exactly the same way.

Enter Embeddings

Instead of representing a word with one number...

Modern AI represents every token using thousands of numbers.

Instead of

it becomes something like

[0.24,
-0.51,
0.84,
...
768 values
]

This is called an Embedding Vector.

Why So Many Numbers?

Because one number cannot describe meaning.

Imagine describing a person.

You might use

Height
Weight
Age
Language
Profession
Education

Each characteristic becomes one dimension.

Now imagine describing every possible relationship between every word ever written.

Suddenly...

768 dimensions doesn't seem so large.

Modern frontier models often use embedding sizes between 2,048 and 16,384 dimensions, depending on the architecture.

Visualizing Embeddings

Imagine shrinking thousands of dimensions into two.

Words with similar meanings naturally cluster together.

The model learned these relationships during training—not because anyone manually programmed them.

Why "King - Man + Woman = Queen" Works

One of the most famous demonstrations of embeddings is vector arithmetic.

This doesn't happen because the model knows royal history.

It happens because the embedding space captures statistical relationships between words.

Later in this series, we'll build these vectors ourselves and visualize them interactively.

The Hidden Geometry of Language

Every token becomes a point inside a gigantic mathematical space.

Modern models don't compare words.

They compare distances between vectors.

Two similar ideas are closer together.

Two unrelated ideas are farther apart.

This geometric representation is one of the key breakthroughs that enabled modern language models.

Before the Transformer Begins

At this point, our sentence has become

The cat sat on the mat.

↓

Token IDs

↓

Embedding Vectors

But one important problem remains.

How does the model know which word came first?

Dog bites man

≠

Man bites dog

The embedding vectors alone cannot answer this.

We need positional information.

That is the focus of the next article.

Architecture So Far

flowchart LR
    %% Nodes
    A([👤 User Prompt])
    B["🔤<br/><b>Tokenization</b><br/><small>Split text into tokens</small>"]
    C["🔢<br/><b>Token IDs</b><br/><small>Convert tokens to IDs</small>"]
    D["🧩<br/><b>Embedding Layer</b><br/><small>Dense Vector Representation</small>"]
    E["📊<br/><b>Vector Space</b><br/><small>Semantic Meaning</small>"]
    F["📍<br/><b>Positional Encoding</b><br/><small>Next Episode ➜</small>"]

    %% Flow
    A -->|Input Text| B
    B -->|Vocabulary Lookup| C
    C -->|Embedding Matrix| D
    D -->|High-Dimensional Vectors| E
    E -->|Coming Next| F

    %% Colors
    style A fill:#2563eb,color:#fff,stroke:#1e3a8a,stroke-width:2px
    style B fill:#22c55e,color:#fff,stroke:#166534,stroke-width:2px
    style C fill:#f97316,color:#fff,stroke:#9a3412,stroke-width:2px
    style D fill:#8b5cf6,color:#fff,stroke:#5b21b6,stroke-width:2px
    style E fill:#06b6d4,color:#fff,stroke:#155e75,stroke-width:2px
    style F fill:#ec4899,color:#fff,stroke:#9d174d,stroke-width:2px

Interactive Components for This Chapter

🎮 Playground 1

Convert text into Token IDs.

🎮 Playground 2

Visualize embedding vectors.

Users can:

Drag words
Zoom clusters
Compare distances
Search vocabulary

🎮 Playground 3

Vector Arithmetic

Live calculation.

💻 Python Example

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")

text = "The cat sat on the mat."

ids = tokenizer.encode(text)

print(ids)

Output

[464,3797,3332,319,262,2603,13]

💻 PyTorch Example

import torch

embedding = torch.nn.Embedding(
    num_embeddings=50000,
    embedding_dim=768
)

token = torch.tensor([3797])

vector = embedding(token)

print(vector.shape)

Output

torch.Size([1,768])

Key Takeaways

✅ Computers understand numbers—not words.

✅ Token IDs are identifiers, not meanings.

✅ Embeddings convert IDs into dense mathematical vectors.

✅ Similar concepts occupy nearby regions in vector space.

✅ Every modern LLM performs reasoning using vectors rather than text.

What's Next?

In Part 3, we'll explore Positional Encoding and answer a deceptively simple question:

If embeddings only represent meaning, how does a Transformer know the difference between "Dog bites man" and "Man bites dog"?

We'll build interactive visualizations of sinusoidal positional encoding, Rotary Position Embeddings (RoPE), and show why word order is critical for modern AI.

Series Progress

✅ Part 1 — What Really Happens When You Ask Claude?
✅ Part 2 — Why AI Sees Everything as Numbers
⏳ Part 3 — Positional Encoding: Teaching AI the Order of Words
⏳ Part 4 — The Mathematics Behind Self-Attention
⏳ Part 5 — Building Your First Tokenizer
⏳ Part 6 — Building Your First Transformer