Transformers and Attention
If you understand the Transformer, you understand the foundation of almost every modern AI model.
By the end of this lesson you will be able to
- explain why RNNs struggled and what the Transformer fixed
- describe attention, self-attention and multi-head attention in plain words
- explain Query, Key and Value with a library analogy
- tell encoder models (BERT) from decoder models (GPT)
This is one of the most important topics in modern AI. Understand Transformers and you understand the foundation of ChatGPT, Gemini, Claude, DeepSeek, Llama, BERT, Amazon Nova, Vision Transformers, and even parts of Stable Diffusion.
A Transformer is a deep learning architecture that understands how the parts of a sequence (like words in a sentence) relate to each other, using a mechanism called attention.
Think of it as the brain inside modern Large Language Models.
Why was the Transformer invented?
Before 2017, language models mostly used RNNs (Recurrent Neural Networks), LSTMs and GRUs. They read one word at a time, left to right, passing a memory along. That works for short sentences, but try this one:
The boy who was wearing a red shirt and carrying a heavy school bag that his grandmother gave him last year went to school because…
By the time you reach the end, the beginning is almost forgotten. RNNs behave exactly like that: information has to travel through every word in turn.
❌ Slow ❌ Forgets long sentences ❌ Can't process words at the same time
The big idea: read everything at once
Ask “What is the capital?” about “The capital of France is Paris.” Your brain jumps straight to France and Paris; you don't reread word by word. Transformers imitate this. Instead of reading sequentially, they look at the whole sentence at once, which makes them much faster and much better at context.
RNN
Each student can only whisper to the next one: A → B → C → D → E. Information passes slowly and gets distorted.
Transformer
Every student can talk to every other student at the same time. Everyone quickly finds the most relevant information.
The superpower: attention
Read: “The cat chased the mouse because it was hungry.” Who was hungry? The cat. Your brain automatically paid more attention to cat and hungry than to mouse. Transformers do the same thing, mathematically.
Attention helps the model decide which words matter most while it processes the current word.
It also resolves double meanings. In “I deposited money in the bank”, attention to money and deposited says “financial bank”. In “I sat on the bank of the river”, attention to river says “river side”.
The weights in this demo are illustrative, chosen to show the idea. A real model computes them from its learned Query and Key vectors.
Write “Rohit hit the ball because he was angry.” Ask: who is “he”? Everyone says Rohit. Nobody consulted a grammar rule; your brain simply connected he → Rohit. Transformers perform a similar operation using attention scores.
How attention works: Query, Key and Value
Imagine every word asks three questions: What am I looking for? What do I contain? What information can I share? These become three vectors, and they are the foundation of self-attention:
Query (Q)
What this word is searching for.
Key (K)
A label describing what each word offers.
Value (V)
The actual information each word carries.
You walk into a library looking for a book on Python: that's your Query. Each book has a title on its spine (Java, Python, C++, AWS): those are the Keys. The content inside each book is its Value. You compare your Query with every Key, find the best matches, and read their Values. In a Transformer, every word does this with every other word, at the same time.
Mechanically: each word's Query is compared with every Key to get a relevance score; the scores become weights (using softmax, so they add up to 1); and the word's new meaning is a weighted blend of all the Values.
Why “self”-attention?
Because the sentence pays attention to itself. In “The dog chased the cat because it was fast”, the word it looks at dog, cat, chased and fast, and computes which are most relevant. Every word examines every other word.
Multi-head attention
One person notices one thing; five people notice five different things. Instead of one attention calculation, a Transformer runs many in parallel, called heads. One head might learn grammar, another meaning, another location, time or relationships. All of them work simultaneously.
Positional encoding
Reading everything at once has a catch: word order is lost. Yet “Dog bites man” and “Man bites dog” mean very different things. So Transformers add a positional encoding to each token, a signal that tells the model where the word sits in the sequence.
The architecture
Encoder vs decoder
The original Transformer, from the 2017 Google paper “Attention Is All You Need”, had two halves: an encoder that reads and understands the input, and a decoder that generates the output. Translation used both: English sentence → encoder → decoder → French sentence. Today many models use just one half.
Encoder models
Read and understand. BERT, RoBERTa.
Used for search, classification, sentiment analysis, named entity recognition.
Decoder models
Generate text one token at a time. GPT, ChatGPT, Claude, Llama, DeepSeek, Gemini (decoder-based or hybrid depending on the model).
Used for chatbots, text generation, coding assistants.
Why Transformers win
| RNN | Transformer |
|---|---|
| Sequential processing | Parallel processing |
| Slow | Fast |
| Struggles with long context | Handles long context much better |
| Limited memory | Attention over the entire sequence |
| Hard to scale | Scales across many GPUs |
| Older architecture | Foundation of modern LLMs |
Where Transformers are used
| Application | Transformer's role |
|---|---|
| ChatGPT | Text generation |
| Gemini | Multimodal AI |
| Claude | Conversational AI |
| GitHub Copilot | Code generation |
| Google Translate | Machine translation |
| Gmail Smart Reply | Text prediction |
| Amazon Bedrock models | Text generation |
| Image generation | Transformer components inside many diffusion models |
| Vision AI | Vision Transformer (ViT) |
| Speech recognition | Audio Transformers |
| Medical AI | Clinical document analysis |
| Autonomous vehicles | Perception and planning |
Fun facts
- Transformers were introduced in 2017 in the Google paper “Attention Is All You Need”.
- GPT stands for Generative Pre-trained Transformer. The T is literally this architecture.
- BERT, GPT, T5, Llama, Claude, Gemini and DeepSeek are all built on Transformer ideas.
- Transformers revolutionised AI because they made large-scale parallel training on GPUs practical.
Key takeaways
- RNNs read one word at a time and forget; Transformers read everything at once.
- Attention lets each word focus on the most relevant other words, using Query, Key and Value.
- Multi-head attention looks for many kinds of relationships; positional encoding keeps word order.
- Encoders understand (BERT); decoders generate (GPT, Claude). The key innovation: parallel processing with attention.
Check your understanding
1. Why do Transformers need positional encoding?
“Dog bites man” and “Man bites dog” contain the same words. Position signals restore order.
2. In the library analogy, what is the Key?
Query = your search, Key = the label, Value = the content you retrieve.
3. Which model type would you choose for sentiment classification?
Encoders are designed to read and understand input, which suits classification. (Decoders can do it too, but encoders are the classic choice.)