RNNs read a sequence one element at a time, carrying a memory forward step by step. That's a natural fit for text, but it creates two real problems: you can't process step 50 until step 49 is done, so training is slow and hard to parallelize.
Worse, information from early in a long sequence tends to fade by the time the network reaches the end — a bit like trying to remember the first sentence of a long conversation. This made RNNs weak at connecting ideas that are far apart in a document.
Knowledge Check
5 questions — answer all, then submit
1. What was the main limitation of RNNs that Transformers overcame?
2. What does multi-head attention add over single-head attention?
3. Why does a Transformer need positional encoding?
4. Which architecture is used by GPT-style large language models?
5. Why does tokenization affect how models handle arithmetic?