LLMs for Engineers — Part 4

So far in this series, we’ve talked about where the data comes from and how text gets converted into tokens.

That gives us the raw ingredients, but it still doesn’t explain the most important part how the model actually learns from all of this and how it ends up generating responses that feel surprisingly intelligent. If you strip everything away, what remains is a very simple objective, but one that becomes incredibly powerful at scale . The model is trained to predict what comes next.

To understand this properly, it helps to change how you think about the data. Instead of imagining documents, web pages, or structured knowledge, it’s better to think of the entire training dataset as one long continuous stream of tokens. There are no real boundaries in the way the model sees it is just sequences of numbers flowing endlessly.

In large systems, this stream can be on the order of trillions of tokens, but conceptually, it’s just a sequence where each token follows another. The model’s job is to observe this stream and learn the patterns that govern how tokens tend to follow each other.

Now imagine we take a small slice of this stream. For example, something like “router sends data packet”.

[router, sends, data, packet]

This slice is what we call the context. The training task is surprisingly straightforward: given this context, predict the next token. In this case, a reasonable continuation might be “forward”.

But instead of directly outputting the word “forward”, the model produces a probability distribution over all possible tokens in its vocabulary. So it might assign a higher probability to “forward”, a slightly lower probability to “drop”, and smaller probabilities to many other tokens. At the beginning of training, these probabilities are essentially random because the model hasn’t learned anything yet.

At this point, it helps to pause and visualize what is actually happening. The entire learning process can be reduced to a simple flow a sequence goes in, probabilities come out, and the model moves toward the correct answer.

If you look at the diagram, you can see exactly what we just described. A sequence of tokens such as “router sends data packet” is fed into the neural network, which then produces probabilities for the next token. The correct answer in this case “forward” is already known from the dataset, and the model is adjusted so that its probability increases. This process is repeated over and over again, across massive amounts of data, gradually shaping the model’s internal parameters.

This is where the learning actually happens. Since we already know the correct next token from the dataset, we can compare the model’s prediction with the actual answer. If the model assigned a low probability to the correct token, we adjust its internal parameters slightly to increase that probability and decrease the probabilities of incorrect tokens. The adjustment itself is very small, almost negligible in isolation, but when you repeat this process across millions of sequences and eventually trillions of tokens, the model begins to capture meaningful structure in the data.

What’s important here is that the model is not learning rules in the traditional sense. It is not explicitly told that routers forward packets or that certain facts must always hold true. Instead, it is learning statistical relationships between tokens based on how frequently they appear together. If a particular sequence shows up often enough, the model encodes that relationship into its parameters. Over time, these relationships become strong enough that the model’s predictions start to look intelligent, even though they are fundamentally based on probabilities.

If we look inside the model, things become more mathematical but not necessarily more mysterious. At its core, the model is simply a function that maps input tokens to output probabilities. Internally, this involves a series of matrix multiplications, additions, and nonlinear transformations applied across many layers. The behavior of this function is controlled by its parameters often billions of them which are adjusted during training. You can think of these parameters as knobs that are gradually tuned so that the model produces better predictions over time.

Once training is complete, the model stops learning and switches to what we call inference, which is what happens when you interact with it. Suppose you type “router sends data”. The model processes this input and predicts a probability distribution for the next token. It might assign high probability to “packet”, but instead of always choosing the highest probability token, the model samples from this distribution. This introduces a controlled amount of randomness, allowing the model to produce more varied and natural responses.

Again, it helps to visualize what is happening during generation. The diagram shows this process clearly. Instead of updating the model, we now repeatedly feed the growing sequence back into the network and sample one token at a time.

Each step looks simple in isolation, but together they form a loop. The model takes the current sequence, predicts the next token, appends it, and then repeats the process. So “router sends data” becomes “router sends data packet”, then “router sends data packet forward”, and so on. What looks like a complete sentence to us is actually built one token at a time through this iterative process.

This also explains many of the behaviors people observe when using models like ChatGPT. Because the model is generating text based on probabilities rather than verifying facts, it can sometimes produce outputs that sound confident but are incorrect. Similarly, because sampling introduces randomness, the same input can lead to slightly different outputs on different runs. These are natural consequences of how the system works, not anomalies.

One useful way to think about the model is as a compressed representation of the data it was trained on. It doesn’t store exact copies of the internet, but it captures patterns in a highly condensed form within its parameters. When generating text, it reconstructs these patterns in new combinations, which is why it can produce both familiar and novel outputs.

If there’s one idea to take away from all of this, it’s that ChatGPT is fundamentally a next-token prediction system operating over sequences of tokens. Everything else the fluency, the apparent reasoning, and the usefulness emerges from this simple mechanism applied at massive scale. Once you internalize this, many of the behaviors of large language models start to make much more sense.

In the next post, we’ll build on this foundation and explore how a base model, which is essentially just a next-token predictor, gets transformed into something that behaves like an assistant — following instructions, answering questions, and interacting in a helpful way. That’s where alignment and post-training come into the picture, and it’s what turns a raw model into something like ChatGPT.

Smiles :)

Anurudh

LLMs for Engineers — Part 3

Welcome back to our LLM Journey. If you are new to here , don’t miss to check previous posts before diving into this post.

So far, we’ve seen:

• ChatGPT predicts the next word

• It learns from internet-scale data

Most of us assume that AI reads text the same way we do. We see words, sentences, and meaning. So it’s natural to think that models like ChatGPT also process language in a similar way.

But that’s not what’s happening at all.

In fact, the model doesn’t see words. It doesn’t see sentences either. What it actually sees is something much more basic numbers.

To understand this properly, it helps to zoom in a bit.

Take a simple sentence like “router sends data”. When you read it, you instantly understand what it means. But a computer can’t work with text directly. The first thing it does is convert everything into a numerical form. At the lowest level, this means turning characters into bytes, and bytes into numbers. So even before we talk about AI, the sentence is already transformed into a long sequence of numbers.

Now here’s the problem. If we keep everything at the character or byte level, the sequence becomes very long and inefficient. And in models like these, sequence length matters a lot. Longer sequences mean more computation, more cost, and slower processing. So we need a better way to represent text.

This is where tokenization comes in.

Instead of working with individual characters, the model groups pieces of text into units called tokens. You can think of tokens as chunks of text that are more efficient to work with. Sometimes a token is a full word, sometimes it’s part of a word, and sometimes it even includes spaces or punctuation.

For example, the sentence “router sends data” might be broken into tokens like:

[router, sends, data]

Each of these tokens is then mapped to a unique number. So what the model actually sees is not the words themselves, but something like:

[1532, 8471, 2983]

These numbers are just IDs. They don’t carry meaning by themselves. They are simply a way to represent text in a format the model can process.

At this point, it might feel like we’re losing information, but we’re actually making things more efficient. By using tokens, we reduce the length of the sequence while still keeping important patterns intact. Common patterns like “data”, “packet”, or “forward” can become single tokens, while rare or complex words can be broken into smaller pieces. This balance is what makes tokenization powerful.

A natural question here is: how does the model decide what becomes a token?

The idea is surprisingly simple. We look at large amounts of text and find patterns that appear frequently. If certain combinations of characters show up again and again, we merge them into a single token. Over time, this builds a vocabulary of tokens that covers common patterns in language. This process is often done using methods like Byte Pair Encoding, but you don’t need to worry about the details. The key idea is that tokens are designed to make text both compact and flexible.

If you want to really understand this, it helps to try it yourself once.

Go to:

👉

https://tiktokenizer.vercel.app/?model=cl100k_base

Type something simple like “router sends data”.

Now try small changes. Add an extra space. Change the case. Add punctuation. You’ll notice that the tokens change. Sometimes even a tiny change in text leads to a completely different token sequence.

This is an important moment.

Because what you type is not what the model sees.

The model only sees tokens.

That means when you change your prompt even slightly ,you are actually changing the sequence of tokens going into the model. And that can change the output.

This explains a lot of things people notice when using AI. Why small wording changes affect responses. Why formatting matters. Why sometimes adding or removing a word changes everything. It’s all happening at the token level.

If you like networking analogies, you can think of this like packetization. Before sending data over a network, we break it into packets. Here, before feeding text into a model, we break it into tokens. The model doesn’t see the original message it sees the encoded form.

So when you write a prompt, you’re not really writing for a human. You’re designing a sequence of tokens for a machine.

And that sequence is what drives everything that comes next.

In the next post, we’ll build on this idea and see what the model actually does with these tokens. Because once text becomes tokens, the next step is where the real magic — or more accurately, the math — begins.

LLMs for Engineers — Part 2

In the last post, we uncovered something surprising:

ChatGPT is just predicting the next word.

When people first hear about ChatGPT, one of the most common questions is this: where does it actually learn all of this from? It can answer questions, explain concepts, write code, and even sound conversational. So it’s natural to assume there must be some kind of structured knowledge base behind it.

But that’s not really how it works.

At its core, the model learns from a massive amount of text data. Not neatly organized knowledge, not curated lessons, but raw text collected from across the internet. This includes things like articles, blogs, forums, documentation, and code. The idea is simple — if you expose a model to enough examples of how language is used, it can start picking up patterns. And once it learns those patterns, it can generate similar text on its own.

But this doesn’t mean the model is just blindly copying the internet. The process is more structured than that.

A large part of this data comes from datasets like Common Crawl, which has been collecting web data for years. You can think of it like a system that starts from a set of websites, follows links, and keeps collecting content along the way. Over time, it builds a huge snapshot of the web. But this raw data is messy. It contains everything —useful content, ads, broken pages, scripts, and a lot of noise.

So before any training happens, this data goes through heavy cleaning.

At this point, it helps to think about what we actually want the model to learn from. If we feed it entire web pages as they are, it will learn patterns from things we don’t care about navigation menus, pop-ups, or random scripts. That’s not useful. So the first step is filtering out bad or irrelevant sources. Websites that are spammy, unsafe, or low quality are removed early on.

Once that is done, the next step is extracting the main content. This means stripping away all the extra parts of a webpage and keeping only the meaningful text. If you’ve ever used a “reader mode” in your browser, it’s a similar idea. We only want the actual content, not the surrounding clutter.

After that comes language filtering. Since most large models are trained primarily on English, the system checks whether the text is mostly in English. If a page doesn’t meet a certain threshold, it might be excluded. This step directly affects what languages the model becomes good at.

Then comes deduplication, which is more important than it sounds. The internet has a lot of repeated content copied articles, mirrored pages, or slight variations of the same text. If the model sees the same thing again and again, it doesn’t learn anything new. So duplicates are removed to make the dataset more useful.

Finally, there is filtering for sensitive information. Things like personal data, phone numbers, or credit card details are removed. This step is important for safety and privacy.

If you step back and look at all this, the process is actually quite logical. We start with a huge amount of raw data, then clean it step by step until we are left with mostly useful text. What remains is not perfect, but it’s good enough for the model to learn patterns from.

Even after all this filtering, the dataset is still enormous. We are talking about tens of terabytes of text. That’s far beyond what any human could read in a lifetime. And that scale is what makes these models powerful. The more data they see, the better they get at capturing how language works.

But here’s an important point that is easy to miss.

The model is not learning facts in the way we think. It is not storing information like a database. Instead, it is learning patterns in how words and sentences are used. If something appears frequently in the data, the model becomes more likely to generate it. If something is rare or inconsistent, the model is less confident about it.

So when you ask a question, the model is not retrieving an answer from memory. It is generating a response based on patterns it has learned from all this data.

That’s a subtle but very important difference.

You can think of this entire process as building the “experience” of the model. Just like humans learn language by reading and listening over time, the model learns by seeing massive amounts of text. The difference is scale ,the model sees far more data than any human ever could.

And this is just the first stage.

Because even after learning from all this data, the model is still not ChatGPT yet. It has learned patterns, but it doesn’t know how to behave like an assistant. That part comes later.

In the next post, we’ll zoom in further and understand how this text is actually represented inside the model. Because before the model can learn patterns, it needs to convert everything into a form it can work with.

And that’s where tokens come in.

Smiles :)

Anurudh

Subscribe now

LLMs for Engineers — Part 1

When you first use ChatGPT, it feels a bit strange.

You ask a question, and it replies almost instantly. Not just with random text, but with something that feels structured, confident, and often surprisingly accurate. It explains things clearly, writes code, fixes errors, and even sounds conversational.

At some point, a natural question comes up.

👉 What is actually going on behind this?

Is it thinking?
Does it understand what I’m saying?
Is there some kind of intelligence inside?

The surprising answer is… not really.

Or at least, not in the way we usually think about intelligence.

What ChatGPT is doing at its core is much simpler than it appears. It is not searching the internet in real time. It is not looking up answers in a database. And it is not reasoning like a human sitting across from you.

Instead, it has learned patterns from a massive amount of text and uses those patterns to generate responses. In fact, one of the simplest ways to describe what it does is this:

👉 It tries to predict what should come next in a piece of text.

That’s it.

Now this might sound almost too simple.

How can something that just “predicts the next word” write essays, explain concepts, or answer questions?

The answer lies in scale.

The model has seen an enormous amount of text articles, documentation, conversations, code, and more. Over time, it has learned how language usually flows. So when you give it a prompt, it doesn’t “think” about the answer. It continues the pattern in a way that looks correct.

You can imagine it like this.

If you had read millions of examples of how people explain something, you would also get very good at continuing those explanations. You might not know the exact source of every fact, but you would still be able to produce something that sounds right.

That’s roughly what’s happening here.

This also explains something important.

Sometimes ChatGPT gives answers that sound very confident… but are not fully correct. That’s because it is not verifying facts. It is generating what is most likely to come next based on patterns it has learned.

So in a way, you are not talking to a system that “knows” things.

You are interacting with a system that is very good at producing language that looks like knowledge.

At this point, things start to look different.

It stops feeling like magic and starts feeling like a system with a very specific behavior.

And once you see it this way, a lot of things begin to make sense:

• why small changes in prompts change the output

• why it sometimes makes mistakes

• why it can sound intelligent without actually understanding

But this is just the surface.

Because now the real questions begin:

👉 Where did it learn these patterns from?
👉 How does it even read text?
👉 What is happening inside when it generates a response?

That’s exactly what we’ll explore in this series.

We’ll break this down step by step, starting from the data, then tokens, then prediction, and finally what’s inside the model itself.

By the end, you won’t just use ChatGPT.

👉 You’ll understand how it works.

Next in this series

👉 Where ChatGPT actually learns from

Smiles :)

Anurudh