Beyond Attention: How Selective State Spaces Match Transformers at Linear Cost
Gu and Dao’s Mamba architecture introduces input-dependent selection into structured state space models—closing the performance gap with Transformers on language, DNA, and audio while scaling linearly in sequence length.
Transformers Are Brilliant—and Brutally Expensive
Nearly every AI system you've heard of—ChatGPT, LLaMA, Gemini—runs on the same basic engine: the Transformer. At its heart is an operation called self-attention, which lets the model look at every other word in a sequence when deciding what comes next. That's powerful. It's also ruinously expensive.
The cost of attention grows quadratically with the length of the input. Double the sequence length and you quadruple the computation. This means Transformers hit a wall: they struggle with very long documents, hours of audio, or entire genomes, because the amount of work (and memory) becomes impractical.
Think of it like a dinner party. Attention insists that every guest have a one-on-one conversation with every other guest before the meal. With 10 people that's manageable (45 conversations). With 1,000 it's half a million conversations, and dinner is getting cold.
Researchers have spent years trying to build cheaper alternatives—linear attention, gated convolutions, recurrent models, structured state space models (SSMs). Many of these scale much better with sequence length. But none of them could actually match the Transformer on the task that matters most: language modeling. They worked well on audio signals and other "smooth" data, but fell short on the kind of discrete, information-dense text that powers large language models.
Gu and Dao identified the root cause: these models are blind to content. Their parameters stay the same no matter what token flows through them. They know about when things happen in a sequence (timing), but they can't reason about what things happen (content). That distinction turns out to be everything.
Selection: Teaching a Model to Pay Attention Without Attention
The central innovation in this paper is what the authors call the selection mechanism. The concept is deceptively simple: let the model's internal parameters change depending on the input.
To understand why this matters, you need to know what a state space model does. An SSM maintains a hidden "state"—a compressed summary of everything it has seen so far—and updates that state as each new token arrives. Prior SSMs used fixed rules for this update. The same mathematical transformation was applied at every step, regardless of what the input actually was. This is called Linear Time Invariance (LTI), and it's a well-studied property from control theory and signal processing.
LTI is great for efficiency. It means you can compute the whole operation as one big convolution (a single parallelizable pass), which is fast. But it's terrible for making decisions. The model can't choose to remember an important token or forget an irrelevant one, because it treats every token identically.
Key Insight — The fundamental problem of sequence modeling is compressing context into a smaller state. Transformers solve this by not compressing at all—they store everything (the "KV cache"), which is effective but expensive. Recurrent models compress into a fixed-size state, which is cheap but only as good as the compression. The selection mechanism makes that compression intelligent: the model learns to keep what matters and discard what doesn't.
The authors demonstrate this with two elegantly simple tasks:
Selective Copying. Imagine a sequence with a few "important" tokens scattered among many irrelevant filler tokens. The model must remember only the important ones and reproduce them later. Previous SSMs could solve a simpler version where the spacing was fixed (just memorize the pattern), but failed when the spacing was random—because identifying which tokens matter requires actually looking at the content.
Induction Heads. This is a pattern-matching task that researchers believe underlies much of the in-context learning ability in large language models. If the model has seen "Harry Potter" earlier in the text, and now sees "Harry" again, it should predict "Potter." This requires the model to associate tokens based on context, not just position—a content-dependent operation that LTI models fundamentally cannot perform.
Mamba solves both tasks cleanly. On induction heads, it generalizes to sequences over 1 million tokens long—4,000× longer than anything it saw during training. No other architecture tested could extrapolate beyond 2×.
The Mechanics: Making Parameters Input-Dependent
Technically, the selection mechanism works by making three key SSM parameters—called Δ, B, and C—functions of the current input, rather than fixed values.
Each of these plays a distinct role:
Δ (Delta) is the most critical. It controls how much the model focuses on the current input versus retaining its existing memory. A large Δ effectively resets the state, saying "forget what came before and pay attention to this." A small Δ means "this input is noise, keep the current state as-is." The paper proves that when you simplify the SSM to its most basic form, this Δ mechanism becomes mathematically equivalent to the gating in classical RNNs like LSTMs and GRUs. So Mamba isn't discarding decades of RNN research—it's providing a principled, continuous-time foundation for what those gates were heuristically doing all along.
B and C give finer-grained control. B governs what gets written into the state (input filtering), while C governs what gets read out from the state (output filtering). Making both selective means the model can modulate its recurrent dynamics based on both the current input and the accumulated context.
Ablation experiments confirm that Δ alone provides the biggest single improvement (about 1.1 perplexity points on language modeling), but combining all three selective parameters synergizes for the best result (about 2.2 points total).
The Engineering Problem—and Solution
Making parameters input-dependent breaks the LTI property, which means the model can no longer be computed as a convolution. That's a serious efficiency problem—convolutions are what made SSMs fast in the first place.
The authors solve this with a hardware-aware algorithm that exploits the memory hierarchy of modern GPUs. The key observation: most operations in the selective scan are memory-bound, not compute-bound. What kills performance isn't the math—it's shuttling data between slow GPU main memory (HBM) and fast on-chip memory (SRAM).
Their solution uses three classical techniques in concert: kernel fusion (do all the steps in one GPU kernel to avoid memory round-trips), parallel scan (a well-known algorithm that parallelizes sequential recurrences), and recomputation (don't store intermediate states for backpropagation; recalculate them on the fly). The result uses the same memory as FlashAttention, runs up to 3× faster than prior SSM implementations on A100 GPUs, and scales truly linearly with sequence length.
The Mamba Block: Radical Simplification
A standard Transformer layer has two distinct sub-blocks: a multi-head attention (MHA) block and a multi-layer perceptron (MLP) block, interleaved with normalization and residual connections. Most prior SSM architectures (like H3) adopted this same two-block design, just replacing attention with an SSM.
Mamba collapses both into a single, unified block. The architecture takes the gated structure of an MLP and embeds the selective SSM directly inside the main processing branch, with a short convolution layer before it. There's no attention module, no separate MLP block—just one homogeneous block, stacked repeatedly.
The design uses an expansion factor of 2 (doubling the internal dimension), and two stacked Mamba blocks match the parameter count of a Transformer's MHA+MLP pair. The activation function is SiLU (Swish), making the gated path equivalent to the popular SwiGLU variant used in LLaMA and PaLM. An optional LayerNorm is added inside the block, borrowed from RetNet.
The result is an architecture that is simpler, has fewer moving parts, and—as the benchmarks show—performs at least as well.
The Numbers: Across Language, DNA, and Audio
Language Modeling
This is the headline result. The authors trained models from 130M to 2.8B parameters on the Pile dataset, following GPT-3 training protocols. They compared against both a standard Transformer (GPT-3 architecture) and a heavily optimized "Transformer++" recipe that incorporates rotary embeddings, SwiGLU, RMSNorm, no linear bias, and higher learning rates—essentially the LLaMA recipe.
Mamba is the first attention-free model to match the Transformer++ scaling curve. At larger sizes and longer sequences, it pulls even further ahead. The gap between Mamba and every other non-attention model (Hyena, RWKV, RetNet, H3) is substantial.
| Model | Params | Avg. Accuracy (7 tasks) |
|---|---|---|
| Pythia | 2.8B | 59.1 |
| RWKV | 3B | 59.6 |
| Mamba | 2.8B | 63.3 |
| GPT-J | 6B | 63.0 |
| Pythia | 6.9B | 61.7 |
Mamba-2.8B doesn't just beat models its own size—it matches or exceeds Transformers with more than twice as many parameters. On common-sense reasoning benchmarks, it scores 4 points higher than Pythia-3B and even edges out Pythia-7B on average.
DNA Sequence Modeling
DNA is a fascinating test case: it consists of discrete tokens (the four nucleotide bases), requires extremely long-range dependencies, and has a massive "vocabulary" of meaningful patterns. The authors used the HG38 human genome dataset (about 4.5 billion base pairs) and compared against HyenaDNA and Transformer++.
Two results stand out. First, Mamba scales more efficiently with model size, matching the other architectures with 3–4× fewer parameters. Second—and more striking—Mamba is the only model whose performance improves monotonically with context length, all the way up to sequences of 1 million tokens. HyenaDNA actually gets worse with longer contexts, presumably because its LTI convolutions aggregate noise along with signal and have no way to filter it out.
On a downstream classification task distinguishing between the five great ape species (which share 99% of their DNA), Mamba reached roughly 75% accuracy at the longest context lengths. HyenaDNA plateaued around 50%.
Audio Waveform Modeling
On audio, the authors slotted Mamba blocks into the SaShiMi U-Net architecture (which previously used S4+MLP blocks) and evaluated on piano music generation and speech generation. Mamba improved across the board, and the improvements grew larger with longer context—consistent with the "better compression" thesis.
On the SC09 speech generation benchmark, a small 6M-parameter Mamba model cut the Fréchet Inception Distance (FID) score to 0.94, beating the previous state of the art (SaShiMi at 1.99) by more than half, and outperforming much larger GAN and diffusion models. A parameter-matched 24M model pushed FID down to 0.67.
Three Mechanical Effects
The authors identify three concrete abilities that selection provides:
Variable spacing. The model can ignore "filler" tokens between meaningful content. In text, this means skipping linguistic noise like "um" or "you know." In DNA, it means focusing on functional regions and skipping vast stretches of non-coding sequence. Mechanically, the gate drives toward zero for irrelevant inputs, preventing them from contaminating the state.
Context filtering. Many sequence models empirically fail to improve—or even get worse—with more context. This is counterintuitive: more context should strictly help. The explanation is that LTI models can't ignore irrelevant context; they aggregate everything equally. Selective models can reset their state at any point, discarding accumulated noise. This is why Mamba's performance improves monotonically with context length while HyenaDNA's degrades.
Boundary resetting. When training on multiple documents packed into a single sequence (a common efficiency trick), Transformers use attention masks to prevent information from bleeding across document boundaries. LTI models have no such mechanism. Selective SSMs can reset at boundaries by driving Δ to a large value, achieving the same effect naturally.
What Matters and What Doesn't
The ablation studies are unusually thorough and reveal several non-obvious findings:
Real vs. complex numbers. Prior SSMs relied on complex-valued states for strong performance on perceptual data (audio, video). The authors find that real-valued SSMs work equally well—or better—on discrete data like text and DNA. They hypothesize this maps to a continuous-discrete spectrum: complex numbers help model smooth signals, while real numbers are better for discrete tokens. The one exception in their experiments is audio, where they switched to complex parameterization.
Initialization barely matters (at scale). Prior work showed that careful initialization (based on the HiPPO theory) was critical for SSM performance. With selective SSMs and sufficient data, even random initialization performs just as well. The structural prior becomes less important when the model can learn content-dependent dynamics from scratch.
State dimension is free lunch—with selection. Increasing the SSM state dimension N from 1 to 16 improves perplexity by over a full point, at the cost of only 1% additional parameters. But this only works when B and C are selective. Without selection, increasing N has essentially zero effect—the model can't make use of a larger state because it can't learn what to put in it.
Ablation Takeaway — The single most impactful design choice in the entire paper is making Δ selective—it alone accounts for the majority of the improvement. Everything else (selective B/C, architecture changes, initialization) provides additional but smaller gains. This is consistent with the theoretical connection to RNN gating, where Δ plays the role of the forget/update gate.
Why Mamba Is Fast Where It Counts
Training speed matters. But for deployed models, inference speed is often the bottleneck—and here Mamba has a structural advantage that no amount of Transformer optimization can fully overcome.
At inference time, autoregressive Transformers must maintain and read from a key-value (KV) cache that grows with every generated token. This means memory usage grows linearly with sequence length, and at high batch sizes, you run out of GPU memory fast.
Mamba has no KV cache. Its recurrent state is fixed-size, regardless of how long the generated sequence is. This means it can run at much higher batch sizes on the same hardware. The practical result: Mamba-6.9B achieves higher throughput than Transformer-1.3B—a model 5× smaller. At the 1.4B scale, Mamba sustains about 1,800 tokens/second on an A100 GPU at batch size 128, while a same-sized Transformer runs out of memory at batch size 16.
What the Paper Doesn't Claim
The authors are refreshingly candid about open questions:
Scale. All experiments are at 3B parameters or below. The largest open-source LLMs at the time were 7B+, and the true frontier was 70B+. Whether Mamba's advantages hold at those scales was explicitly left unanswered.
The continuous-discrete trade-off. Selection helps on discrete data (text, DNA) but slightly hurts on continuous signals where LTI models already excelled. This is a genuine architectural trade-off, not a pure win.
Ecosystem and affordances. Transformers have a massive ecosystem: fine-tuning recipes, prompting strategies, RLHF pipelines, quantization tools, in-context learning theory. Whether SSMs support all of these is unknown. The paper asks the question explicitly but doesn't answer it.
Engineering at scale. Scaling SSMs may require engineering adjustments not discussed in the paper. The parallel scan is work-efficient in theory, but practical GPU utilization at very large model sizes is a different challenge.
What Mamba Means for AI Architecture
Mamba's contribution isn't just another model. It's a proof of concept for a different computational primitive: that you can match Transformer-quality language modeling without attention, using a mechanism rooted in classical signal processing and control theory, running in linear time.
The intellectual lineage runs from Kalman filters (1960) through RNNs (1990s) through structured state space models (2021–2022) to selective SSMs (2023). Each step added something: S4 added efficient long-range modeling; Mamba added content-aware selectivity. The result is a model that combines the expressiveness of attention with the efficiency of recurrence—and does so by identifying the precise missing ingredient (input-dependent parameters) rather than bolt-on heuristics.
Whether Mamba—or something descended from it—ultimately replaces the Transformer is an empirical question that depends on scaling, ecosystem, and engineering realities far beyond any single paper. But this work establishes that the design space for foundation model backbones is far wider than "Transformer or bust." And that alone is a significant shift.
The Transformer said: "I will look at everything, all the time." Mamba says: "I will choose what to remember." The question for the field is whether that choice scales as well as brute force. The early evidence says yes.
Based on arXiv:2312.00752 by Albert Gu & Tri Dao — Originally published December 2023, revised May 2024