How Does AI Actually Think? My Q&A with Claude – Part 2

Recap

In Part 1, I asked Claude how a model that only predicts the next word can keep track of meaning. We covered how tokens become vectors, how attention and the MLP keep repositioning them, and how the same machinery handles maths, code and tools. In this part, we look at how the model learns, and then take a closer look at attention.

How it learns

So gradients come in during training, for positions and refinements?

Claude: Yes, with one precision: gradients do not set positions directly. They adjust the weights, which determine positions.

• The embedding table sets each token's starting vector, so its initial position.
• Attention and MLP weights set the rules for how vectors move, so the refinements.

During training the model predicts, compares its guess with the actual next token, measures the error, and the weights are nudged slightly to reduce it. Billions of repetitions shape good starting positions and good movement rules. After training the weights are frozen, and no gradients run when it answers you.

Analogy: training writes the credit policy, refined by past defaults. Answering applies the policy to a new application. The policy decides the outcome, but it is not rewritten for each case.

(MLP stands for multilayer perceptron. It’s the second half of every layer in the model, right after attention, and is also called the feed-forward network.)

What it does: once attention has let each token gather context from the others, the MLP works on each token on its own and transforms its vector. Much of the model’s stored knowledge seems to live here.

Then the gradients come from the MLP?

Claude: No. Gradients start from the error and are calculated by tracing back through every layer. The layers don't produce them; they receive the updates.

1. The model predicts the next token.
2. Loss measures how wrong it was.
3. Backpropagation traces back from that error through every layer (output, MLPs, attention, embeddings), working out how much each weight contributed.
4. Each weight is nudged in the direction that reduces the error.

Analogy: a budget variance report. The gap between budget and actual is the source. Tracing it back to each department tells you where to adjust. Departments are where corrections land, not where the variance comes from.

So the MLP is the variable that adjusts the gradient?

Claude: Reverse it: the MLP's weights are the variables the gradient adjusts. The roles, plainly:

• Loss measures the error.
• Gradient shows how the error rises as each weight changes: it points uphill.
• Optimizer is a separate training algorithm that steps each weight the opposite way, downhill.
• MLP, attention and embedding weights are what get updated.

The chain runs one way: error → gradient → optimizer → weights. Think of y = f(x): the gradient tells you how y (the error) changes as x (the weights) changes, and you move x the opposite way to make y smaller. The MLP holds some of the x's.

What is an optimizer, and how does it know which way to tweak weights?

Claude: It is a simple formula written by engineers, not learned. And it never knows the correct number. It only knows which direction is downhill.

new weight = old weight − step size × gradient

Picture standing on a hillside in thick fog. Height is error and you want the valley. You cannot see the bottom, but you can feel the slope under your feet. That slope is the gradient. Take a small step downhill, feel again, step again. Repeat enough times and you end up somewhere low.

• Adam (and its common variant AdamW) adds two refinements: momentum, so noisy steps do not throw it off, and a separate step size for each weight.
• The learning rate (step size) is set by engineers. Too big and you overshoot the valley; too small and training takes forever.

There is no "right answer" for any single weight. Good weights are simply where the model ends up once the error is low enough.

What is inference, and where does it fit?

Claude: Inference is using the trained model to produce answers. Training builds the weights; inference runs your words through those frozen weights, one token at a time.

The term matters because API pricing per token is inference cost, and "thinking longer" before answering means spending more inference, not retraining. The name comes from statistics, though in machine learning it has come to mean using a fitted model on new data. Training builds the credit-scoring model; inference scores each new applicant.

Attention up close

What is attention? How do tokens "look" at each other?

Claude: "Looking" is just arithmetic. Each token compares itself with the others, scores how relevant each one is, then blends in information from the most relevant.

Each token makes three versions of itself:

• Query: "What am I looking for?"
• Key: "What do I contain?"
• Value: "What will I hand over if picked?"

Take "The bank approved the loan to the company because it had strong cash flows."

1. Score. "It" compares its query with the key of itself and every earlier token. A high score means relevant.
2. Convert to percentages. Scores become shares adding to 100%: say "company" 70%, "loan" 20%, others 10% (illustrative numbers, not measurements).
3. Blend. Take 70% of "company's value, 20% of "loan"'s, and so on.
4. Update. Add that mix to "it"'s vector. Now "it" carries the meaning of "the company".

Every token does this at the same time, and each layer runs many attention "heads" in parallel, each tracking different patterns: grammar, references, topic. Tokens only look backward, since the future has not been written yet. Nobody told the model that pronouns look at nouns; training tuned it until that pattern emerged.

Analogy: a meeting. Each person has a question (query), a name badge describing their expertise (key) and notes to share (value). You scan the badges, listen most to those that match your question, and take mostly their notes.

So it is vector maths with three variables to compare?

Claude: Essentially, but only two of the three are compared. Query and key are compared to produce the relevance score. Value is not compared; it is the content passed along once the scores decide how much to take. Each is a list of many numbers, not a single variable, and it is all matrix multiplication, which is why GPUs run AI so well.

So value is the output, and query and key are the inputs?

Claude: Close. Query and key decide how much to take from each token. Value is what gets taken. The output is the weighted mix of values, not the value itself.

Think of a portfolio: query and key set the allocation percentages, the values are the assets, and the output is the blended portfolio.

By "learned tables", you mean tables used in vector multiplication?

Claude: Yes. They are matrices, grids of numbers.

• Each attention head has three: one each for query, key and value.
• A token's vector multiplied by the query matrix gives its query vector; the same for key and value.
• The numbers in these matrices are weights, set in training by the gradients and optimizer, then frozen.
• Every token in that layer uses the same three matrices. Different tokens get different results because their vectors differ.

In banking terms: one fixed scoring formula applied to every loan application. Same formula, different inputs, different scores.

The whole process in 17 steps

Putting every answer together, here is the full journey: steps 1–8 are training, steps 9–17 are answering you.

1. Start blank. The model begins as billions of random numbers (weights), like a new chef who knows nothing.
2. Read and guess. It reads huge amounts of books, websites, code and maths, guessing the next word (strictly, the next token) each time.
3. Measure the mistake. It compares each guess to the real next word; the size of the error is the loss. The real next word comes from the text itself, so pretraining needs no labelled data.
4. Trace the error back. Working backward from the error (backpropagation) gives the gradient: which way, and how much, each weight pushes the error. The gradient points toward more error, uphill. Gradients are calculated from the error by tracing back through the model's maths; the MLP doesn't produce them, its weights receive the updates.
5. Step downhill. The optimizer, a formula written by engineers, updates every weight: new weight = old weight − step size × gradient. Like walking down a foggy hill.
6. Repeat an enormous number of times. Trillions of guesses, millions of updates. Ideas like "country", "money" or "variable in code" form on their own, as directions on a giant map of meaning.
7. Get coached. Further training teaches chatting, reasoning, coding and tool use. Correct answers and passing code are rewarded, and human raters steer it toward helpful, honest and safe answers.
8. Freeze. The weights are locked. Everything after this is inference.
9. Gather the context. Instructions, tool descriptions and the whole conversation are bundled. This is how it "remembers": everything in the conversation is available to it, up to a size limit.
10. Chop into tokens. Text, code and numbers are split into small pieces.
11. Turn tokens into vectors. Each token gets a starting spot on the meaning map, and the model also keeps track of where in the sequence it sits ("dog bites man" ≠ "man bites dog").
12. Refine through many layers. In each layer:
    • Attention: every token is viewed through three fixed lenses (matrices) to make a query, a key and a value. Queries are scored against keys, scores become percentages, and each token blends in the matching values.
    • MLP: each token's representation is transformed on its own, drawing on learned knowledge.
    • Each layer, with its own weights, nudges every token's spot until it sits where its meaning in this conversation belongs.
13. Predict the next token. The last token's final spot becomes a probability for every token in the vocabulary, tens or hundreds of thousands of them. One is picked, either the top choice or a sample from the likely ones depending on settings, which is why the same question can get different answers.
14. Use a tool if needed. If the model writes a tool request (search, calculator, run code), software outside the model runs it, adds the result to the context and answering resumes.
15. Add and loop. The new token is appended and steps 11–14 repeat, one token at a time. Earlier work is cached.
16. Maths and code work the same way. They are tokens too. Writing steps out acts as a scratchpad; running real code catches mistakes.
17. Stop. It ends when it predicts a special end-of-answer token, picked like any other, or when it hits a length limit.

In one line: training shapes the weights by stepping downhill on error; answering runs your words through those frozen weights to build meaning, then predicts one token at a time.

The best part of the journey was seeing my understandings proved wrong at each step. Each correction was a small step down the foggy hill, which, it turns out, is exactly how the model learns too.