How Claude, ChatGPT, and Gemini Actually Work: Inside a Large Language Model (Part 1)

Part 1 : Building Your Own Foundation Model (Part 1)

What Really Happens When You Ask Claude a Question?

An engineer's guide to understanding the complete lifecycle of a Large Language Model—from your keyboard to billions of mathematical operations happening inside modern AI systems.

---

Imagine typing a single sentence into Claude.

Explain Quantum Computing in simple words.

Within only a few hundred milliseconds, one of the world's most sophisticated software systems begins executing.

That seemingly simple question is transformed into numbers, mathematical vectors, matrix multiplications, attention calculations, probability distributions, and billions of floating-point operations running across thousands of GPUs.

• No dictionary lookup.
• No search engine.
• No database query.

Instead, Claude predicts the most likely next token repeatedly until it forms a coherent answer.

Understanding this pipeline is the first step toward building your own Large Language Model.

In this series, we won't simply use AI.

We'll engineer one.

---

The Journey of One Prompt

flowchart TD

A["⌨ User Types Prompt"]

A-->B["🔤 Step 1<br/>Tokenization"]

B-->C["🔢 Step 2<br/>Convert to Token IDs"]

C-->D["🧩 Step 3<br/>Embedding Vectors"]

D-->E["📍 Step 4<br/>Add Position Information"]

E-->F["🧠 Step 5<br/>Transformer Layers"]

F-->G["🎯 Self-Attention"]

G-->H["⚙ Feed Forward"]

H-->I["📐 LayerNorm + Residual"]

I-->J["📊 Predict Next Token"]

J-->K["🎲 Choose Best Token"]

K-->L{"More Tokens Needed?"}

L--Yes-->F

L--No-->M["💬 AI Response"]

This entire pipeline executes for every generated token, not just once.

If the final answer contains 250 output tokens, the model repeats the prediction process around 250 times, each time considering all previously generated context.

---

Why AI Doesn't Understand Language

This is perhaps the biggest misconception about Large Language Models.

AI does not understand English, Gujarati, Japanese, or any human language in the way people do.

Instead, it converts everything into numbers.

For example:

Hello

might become

[15496]

while

Foundation

could become

[48722]

These numbers are called token IDs.

To the model, language is simply a sequence of numerical identifiers.

Meaning emerges only after these numbers are transformed into high-dimensional vectors and processed through layers of neural computation.

---

Step 1 — Tokenization

Human sentence:

Artificial Intelligence is amazing.

Tokenizer output:

Artificial
Intelligence
is
amazing
.

Converted IDs:

4821
19283
339
9217
13

Modern tokenizers often split unfamiliar words into smaller subword pieces rather than storing every possible word individually.

This dramatically reduces vocabulary size while allowing models to represent virtually any text.

---

Step 2 — Embeddings

Token IDs themselves have no semantic meaning.

The embedding layer maps each token into a dense vector of floating-point values.

Conceptually:

4821

↓

[0.183,
-0.442,
0.972,
...
]

For modern frontier models, each embedding may contain several thousand dimensions.

These vectors capture statistical relationships learned during training.

Words used in similar contexts tend to occupy nearby regions of this high-dimensional space.

---

Step 3 — Position Matters

Consider these two sentences:

Dog bites man.

Man bites dog.

The same words appear, yet the meanings are entirely different.

Transformers solve this by encoding the position of every token.

Without positional information, a model would treat both sentences almost identically.

Later in this series, we'll explore techniques such as sinusoidal position encodings and Rotary Position Embeddings (RoPE).

---

Step 4 — The Transformer

The Transformer architecture changed artificial intelligence forever.

Instead of processing text one word at a time like earlier recurrent neural networks, a Transformer processes the entire sequence in parallel and uses attention to determine which tokens are most relevant to one another.

Each Transformer block generally contains:

• Multi-Head Self-Attention
• Feed-Forward Network
• Residual Connections
• Layer Normalization

Modern LLMs stack dozens—or even hundreds—of these blocks.

---

Step 5 — Self-Attention

Suppose the prompt is:

The cat chased the mouse because it was hungry.

What does "it" refer to?

A Transformer doesn't rely on grammar rules.

Instead, the attention mechanism computes relationships between every token in the sequence and learns that, in this context, "it" is more strongly associated with "cat" than with "mouse."

This ability to model long-range relationships is one of the key innovations behind modern language models.

---

Step 6 — Prediction

After all Transformer layers complete, the model produces a probability distribution over its vocabulary.

A simplified example:

Token	Probability
is	42%
was	21%
can	10%
will	8%
has	5%
Others	14%

The model selects the next token using a decoding strategy such as greedy decoding, top-k sampling, or nucleus (top-p) sampling.

This selection process repeats until an end-of-sequence token is generated or another stopping criterion is met.

The Vector Analogy Example

The Attention Heatmap Example

---

The Hidden Scale

While the pipeline appears straightforward, the scale is extraordinary.

A frontier model may involve:

• Hundreds of billions of parameters
• Trillions of training tokens
• Thousands of GPUs
• Petabytes of training data
• Massive distributed storage systems
• High-bandwidth GPU interconnects
• Sophisticated checkpointing and fault tolerance

The elegance of the architecture lies in the fact that the same prediction loop scales from tiny research models to the world's largest foundation models.

---

What This Series Will Cover

Over the coming parts, we'll build the knowledge required to understand—and eventually implement—every major component of a modern foundation model.

Topics include:

1. The mathematics behind neural networks
2. Tokenization algorithms
3. Vector embeddings
4. The Transformer architecture
5. Self-attention from first principles
6. Training loops and optimization
7. GPU architecture and CUDA
8. Distributed training
9. Mixture of Experts (MoE)
10. Reasoning models
11. Retrieval-Augmented Generation (RAG)
12. AI agents and tool use
13. Quantization and inference optimization
14. Building an end-to-end chat application
15. Future directions, including quantum and hybrid quantum-classical AI research

---

Key Takeaways

• Large Language Models operate on numbers, not words.
• Tokenization converts text into numerical IDs.
• Embeddings transform IDs into meaningful vector representations.
• Positional encoding preserves word order.
• Transformer blocks use self-attention to model relationships across the entire context.
• Responses are generated one token at a time through repeated probability prediction.
• The same fundamental architecture underpins both small research models and today's largest AI systems.

---

What's Next?

In Part 2, we'll leave the software layer and dive into the mathematics that makes modern AI possible.

We'll explore vectors, matrices, tensors, probability, gradients, loss functions, and why matrix multiplication has become the most important operation in contemporary computing.