Transformers, from the inside · Chapter 1
The Residual Stream
By Tomáš Sikora · June 3, 2025
A transformer isn't really a stack of separate layers passing numbers along. It keeps one shared running state — the residual stream — that every block reads from and adds to, and its final value is the prediction.
This chapter builds that state from scratch and then reads it six different ways, as a small model retrieves a fact and computes a sum. The values in the figures are precomputed from a run of Qwen3-1.7B (28 layers, d=2048).
Watch the sum become the answer
The answer emerges across depth, read with the tuned lens.
Meaning is a direction
The embedding space already clusters words by meaning.
The answer is a direction too
Each layer's additive share of the final prediction.
Superposition: more features than dimensions
The model represents more features than it has dimensions.
The anatomy of the stream
Giant outlier dimensions and iterative refinement.
From reading to causing
Activation steering and causal tracing.
This is an interactive article — the figures are explorable and require JavaScript to load.