How Generative AI Models Work: Tokens, Transformers and Embeddings (AI-901)

Generative AI creates new content (text, code, images, audio) instead of only classifying or predicting a number. On AI-901 you need to explain, in simple terms, how these models work. You don't need the maths, but you do need the vocabulary.

From machine learning to generative AI

All of these are related:

• Artificial intelligence (AI): software that imitates human abilities such as understanding language or recognising images.
• Machine learning (ML): models that learn patterns from data instead of following hand-written rules.
• Deep learning: machine learning with many-layered neural networks.
• Generative AI: deep learning models that generate new content. Large language models (LLMs) are the best-known example.

Step 1: text becomes tokens

Models don't read words. Text is split into tokens: whole words, parts of words or punctuation. In English, a token averages about four characters, or roughly three quarters of a word.

"Unbelievable results!" might become tokens like Un believ able results !.

Tokens matter because:

• Models have a context window, a limit on how many tokens of prompt plus answer they can handle at once.
• Pricing for most models is per token, counted separately for input and output.

A quick way to estimate tokens is to count words and multiply by about 1.3. Our word counter gives you the word count.

Step 2: tokens become embeddings

Each token is turned into an embedding: a long list of numbers (a vector) that represents its meaning. Words with similar meanings get similar vectors, so "king" and "queen" sit close together, and both are far from "banana".

Embeddings aren't just an internal detail. Embedding models such as text-embedding-3-small produce vectors for whole sentences or documents, which power semantic search: finding text by meaning rather than exact keywords.

Step 3: the transformer and attention

Modern LLMs use the transformer architecture. Its key idea is attention: for every token, the model works out how much each other token in the context matters.

In "The bank raised its interest rate", attention helps the model link "bank" with "interest rate", so it understands a financial bank, not a river bank.

The original transformer had an encoder (understands input) and a decoder (generates output). GPT-style chat models are decoder-based and generate text.

Step 4: predicting the next token

An LLM generates text one token at a time. At each step it calculates a probability for every possible next token, picks one, adds it to the text, and repeats.

Settings such as temperature control how adventurous that choice is. Low temperature picks the most likely tokens (focused, repeatable). Higher temperature allows less likely tokens (more varied and creative). The next lesson covers parameters in detail.

This explains a key limitation: the model produces plausible text, not verified facts.

How models are trained

1. Pre-training: the model reads a huge amount of text and learns to predict the next token. This is where its general knowledge and language skills come from.
2. Fine-tuning: further training on smaller, task-specific examples, for example to follow instructions or use a company's style.
3. Alignment: techniques such as reinforcement learning from human feedback teach the model to be helpful and avoid harmful responses.

The model's knowledge stops at its training cutoff unless you give it newer information in the prompt or through tools.

Hallucinations and grounding

A hallucination is a confident answer that is wrong or made up. It happens because the model generates likely-sounding text.

The main fix is grounding: giving the model trusted information to answer from. The most common pattern is retrieval-augmented generation (RAG):

1. The user asks a question.
2. The app searches your data (often with embeddings and vector search) for relevant passages.
3. Those passages are added to the prompt.
4. The model answers using that context, ideally with citations.

Types of generative models

Many image generators use diffusion: they start from random noise and remove it step by step until an image matching the prompt appears.

Copilots and agents

Two terms you'll see often:

• A copilot is an AI assistant built into an app that helps a person do a task, such as drafting an email.
• An agent goes further: it has instructions and tools, so it can plan steps and take actions, like looking up an order and creating a refund.

Key takeaways

• Text → tokens → embeddings → transformer with attention → next-token prediction.
• The context window limits tokens per request; pricing is usually per token.
• LLMs generate plausible text, so they can hallucinate. Grounding and RAG reduce this.
• Know the model types: LLM, SLM, reasoning, multimodal, image generation and embedding.

Next lesson: choosing and deploying models in Foundry.