Transformer

Transformers are a family of neural network architectures built around the self-attention mechanism, introduced by Vaswani et al. in the 2017 paper Is All You Need. They have become the dominant architecture for natural language processing and increasingly for computer vision, speech, and multimodal tasks.

Overview

A transformer processes a sequence of tokens by repeatedly mixing information across positions using rather than recurrence or . Each layer applies multi-head self- followed by a position-wise feed-forward network, with and layer normalisation at each sublayer. Because is permutation-invariant, positional information is injected via .

Unlike recurrent neural networks, transformers compute all positions of the sequence in parallel during training, which dramatically improves throughput on modern hardware. This parallelism, combined with 's ability to model long- dependencies in a constant number of layers, has enabled the training of very large models on very large corpora — the foundation of contemporary large (LLMs).

Key Concepts

• Self- — every token in a sequence computes a weighted sum of all other tokens, where the weights are learned from content similarity rather than position.
• Queries, keys, and values — each token is projected into three : queries match against keys to produce weights, which are then used to combine values.
• Multi-head — several operations run in parallel with independent projections, allowing the model to attend to different aspects of the input simultaneously.
• — sinusoidal or learned added to token so the model can distinguish positions in an otherwise order-agnostic operation.
• and layer normalisation — wrap every sublayer to stabilise gradients and enable very deep stacks.
• Feed-forward networks — two-layer position-wise applied independently to each token, providing per-position nonlinear transformation.
• Causal masking — in , future positions are masked out of the so the model cannot peek ahead during autoregressive generation.

History

as a soft alignment between and hidden states was introduced for by Bahdanau et al. (2015) and refined by Luong et al. (2015), but these models were still built on top of RNNs. In 2017, Vaswani et al. proposed the transformer, which removed recurrence entirely and relied solely on . The architecture won the WMT'14 English–German and English–French translation benchmarks while training in a fraction of the time.

In 2018, Devlin et al. released , a deep bidirectional pre-trained with masked-language modelling, which set new state-of-the-art results on a wide of NLP benchmarks. The same year, Radford et al. introduced , a -only autoregressive transformer trained with a standard language-modelling objective. Subsequent scaling — -2 (2019), -3 (2020), PaLM, , and contemporary frontier models — established that transformer performance follows predictable scaling laws (Kaplan et al. 2020) as parameters, data, and compute grow.

Transformers also expanded beyond text: (Dosovitskiy et al. 2021) treat patches as tokens, Speech transformers (e.g. Whisper) operate on audio spectrograms, and multimodal models such as and Flamingo unify several modalities in a single architecture.

Key Approaches

The core building block is scaled . Given of queries $ Q \in \mathbb{R}^{n \times d_k} $, keys $ K \in \mathbb{R}^{n \times d_k} $, and values $ V \in \mathbb{R}^{n \times d_v} $:

$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^{\!\top}}{\sqrt{d_k}}\right) V $

The scaling factor $ \sqrt{d_k} $ prevents the from growing large and pushing the softmax into low-gradient regions. Multi-head runs $ h $ parallel heads with independent projections and concatenates their outputs:

$ \mathrm{MultiHead}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\, W^O $

where $ \mathrm{head}_i = \mathrm{Attention}(Q W_i^Q,\, K W_i^K,\, V W_i^V) $.

Three architectural variants dominate practice:

• – transformers — the original design, used for tasks such as translation and summarisation. layers use bidirectional self-; layers use causal self- and cross- to outputs. Examples: , , the original Vaswani model.
• -only transformers — discard the and use bidirectional self- throughout. Trained with masked-language modelling for . Examples: , , .
• -only transformers — discard the and use causal self- only, trained with next-token prediction. Now the dominant design for general-purpose . Examples: family, , , .

The original transformer uses :

$ \mathrm{PE}(pos, 2i) = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \quad \mathrm{PE}(pos, 2i+1) = \cos\!\left(\frac{pos}{10000^{2i/d}}\right) $

Subsequent work introduced learned absolute positions (, -2), (Shaw et al. 2018, ), rotary position (, Su et al. 2021), and linear biases (Press et al. 2022), the latter two improving extrapolation to longer sequences than seen during training.

Self- has $ O(n^2 d) $ complexity, which is expensive for long sequences. Efficient transformer variants reduce this cost: Linformer and use low- or approximations; Longformer and BigBird mix local and global patterns; (Dao et al. 2022) reorders the computation to be IO-aware, achieving exact with much higher hardware utilisation. Sparse () routing replaces dense feed-forward sublayers with sparsely-activated experts (, ) to scale parameters without proportionally scaling compute.

Connections

Transformers are deeply connected to other ideas in modern machine learning. They build directly on Attention Mechanisms, generalising the soft alignment used in earlier models. Their inputs are typically dense word embeddings (or learned subword such as and ), and their outputs over a vocabulary are produced by a softmax over linear .

Training relies on the same machinery as other deep networks: backpropagation through the and feed-forward sublayers, optimisation by gradient descent variants such as and , and cross-entropy loss for next-token prediction or masked-token recovery. techniques including dropout (on weights and feed-forward ) and weight decay are standard.

Architecturally, transformers can be viewed as a of fully-connected layers with content-conditioned routing, or as a special case of on fully-connected graphs. They are usually pre-trained on large unlabelled corpora and then adapted to downstream tasks via transfer learning — , parameter-efficient methods such as , , or simply prompting in-context.

See also

• Attention Mechanisms
• Neural Networks
• Recurrent Neural Networks
• Word Embeddings
• Softmax Function
• Transfer Learning
• Backpropagation
• Cross-Entropy Loss

References

• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł. and Polosukhin, I. (2017). " Is All You Need". NeurIPS.
• Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019). ": of Deep Bidirectional Transformers for Language Understanding". NAACL.
• Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I. (2018). "Improving Language Understanding by Generative Pre-Training". OpenAI technical report.
• Brown, T. et al. (2020). "Language Models are Few-Shot Learners". NeurIPS.
• Kaplan, J. et al. (2020). "Scaling Laws for Neural Language Models". arXiv:2001.08361.
• Dosovitskiy, A. et al. (2021). "An is Worth 16x16 Words: Transformers for Recognition at Scale". ICLR.
• Su, J., Lu, Y., Pan, S., Murtadha, A., Wen, B. and Liu, Y. (2021). "RoFormer: Enhanced Transformer with Rotary Position ". arXiv:2104.09864.
• Dao, T., Fu, D. Y., Ermon, S., Rudra, A. and Ré, C. (2022). ": Fast and Memory-Efficient Exact with IO-Awareness". NeurIPS.