A 7-Million Parameter Model Just Embarrassed the Giants

What if the secret to better AI reasoning isn't building bigger models, but teaching tiny ones to think harder?

TL;DR: A new model called TRM uses a single 2-layer neural network with 7 million parameters. By applying that network recursively, like editing a draft over and over, it outperforms models with 100,000× more parameters on hard puzzle benchmarks. No massive GPU clusters required.

The Scaling Assumption

Modern AI has been dominated by a simple belief: bigger is better. Want a smarter model? Add more parameters. More data. More compute. Neural scaling laws have formalized this, and for a wide range of tasks, it holds.

But there's a category of problems where even the largest models fall flat: structured reasoning tasks like solving Sudoku puzzles, navigating mazes, or cracking the abstract visual puzzles in ARC-AGI. DeepSeek R1 (671 billion parameters) scores zero percent on Sudoku-Extreme. Not low. Zero. Even with chain-of-thought prompting and test-time compute scaling, frontier LLMs cannot make a dent.

A new paper from Samsung's AI lab in Montréal asks a different question: what if we made the model smaller, but let it think longer?

The Core Idea: Think, Don't Grow

The Tiny Recursive Model (TRM) works on a principle anyone who's written an essay understands: your first draft is never your best.

Instead of making a single pass through a massive network, TRM takes a tiny 2-layer network and runs it over and over on the same problem. Each pass refines the previous answer. The first attempt might be rough. The second fixes obvious errors. By the sixteenth pass, the answer is often correct.

TRM builds on the Hierarchical Reasoning Model (HRM) by Wang et al., which first showed that recursive small networks could outperform LLMs on puzzle tasks. An independent analysis by the ARC Prize Foundation found that HRM's performance was driven primarily by its "deep supervision" training loop, not its complex two-network hierarchy. TRM takes that insight and strips everything else to the minimum.

The Results

On ARC-AGI-1, a benchmark of abstract visual puzzles designed to test genuine reasoning, TRM with 7 million parameters beats Gemini 2.5 Pro, o3-mini, Claude 3.7, and DeepSeek R1. On Sudoku-Extreme, it scores 87.4% where every frontier LLM scores zero.

Method | Params | Sudoku | ARC-1 | ARC-2DeepSeek R1 | 671B | 0.0% | 15.8% | 1.3%Claude 3.7 | ~8K? | 0.0% | 28.6% | 0.7%o3-mini | ? | 0.0% | 34.5% | 3.0%Gemini 2.5 Pro | ~32K? | 0.0% | 37.0% | 4.9%HRM | 27M | 55.0% | 40.3% | 5.0%TRM-Att | 7M | 74.7% | 44.6% | 7.8%TRM-MLP | 5M | 87.4% | 29.6% | 2.4%

Sudoku = Sudoku-Extreme test accuracy. ARC = public eval, 2 attempts. TRM-Att uses self-attention; TRM-MLP replaces it with an MLP.

The MLP variant (TRM-MLP) excels on fixed-length tasks like Sudoku's 9×9 grid but fails on variable-length tasks. The attention variant (TRM-Att) is clearly superior on larger 30×30 grids like ARC-AGI and Maze-Hard.

Architecture: What TRM Actually Does

HRM used two separate 4-layer Transformer networks with a complex biological justification about how the brain processes information at different speeds. TRM throws all of that out and replaces it with something much simpler.

The three things TRM keeps track of

At any point during processing, TRM holds three pieces of state. First, the question x: the original input, embedded once and held constant. Second, the current answer y: the model's best guess so far. Third, a reasoning scratchpad z: internal working memory that cannot be decoded into a readable answer on its own, but helps the model figure out what the answer should be.

If you remove y, the model forgets what it was predicting. If you remove z, it forgets how it got there. Both are necessary; experiments confirm that using fewer or more variables hurts performance.

One network, two jobs

HRM needed two separate networks: one for updating the scratchpad, one for updating the answer. TRM uses a single 2-layer Transformer for both. The trick is simple: when the question x is included in the input, the network knows it should update the scratchpad z. When x is left out, the network knows it should update the answer y. The presence or absence of the question acts as a task switch. No extra weights needed.

The recursion loop (inner): think, think, think... then write

Say n = 6. The model first runs 6 passes of "thinking": each pass feeds the question x, current answer y, and scratchpad z into the network, and gets back an updated scratchpad. The answer stays untouched during this phase. Only after those 6 thinking passes does the model run one "writing" pass: it feeds just the answer y and scratchpad z (no question) into the same network, and gets back an updated answer. That full cycle of 6 thinks + 1 write = 7 network calls is one "recursion block."

Now here's the efficiency trick. The model runs this block T times (default T = 3). But it doesn't need to train through all 3 runs. The first T-1 = 2 runs execute with gradients turned off (torch.no_grad()), which is fast and uses no training memory. Only the final run tracks gradients. The gradient-free runs cheaply improve y and z to a better starting point, then the model learns from the last run where gradients flow through all 7 calls. This completely sidesteps HRM's reliance on the Implicit Function Theorem: instead of assuming recursion has converged to a fixed point and only training the last call, TRM trains through a full block of 7 calls honestly.

The supervision loop (outer): hand it back and try again

Everything above produces one improved answer. But one improvement may not be enough. So the model does it again, up to 16 times. Each time, it takes the answer y and scratchpad z from the previous round, cuts them loose from the computation graph (detach()), and uses them as the starting point for a fresh round of recursion. After each round, the model is scored against the correct answer using cross-entropy loss, and the weights are updated.

This is "deep supervision." The model is not trained to solve the puzzle in one shot. It is trained to take whatever partial answer it currently has and make it a little better. That is a much easier task to learn. Do it 16 times in sequence, and those small improvements compound into a correct solution. The detach() between rounds is what makes this memory-efficient: each round only backpropagates through itself, not through all previous rounds.

Analogy: A student takes an exam. The teacher marks it and hands it back. The student re-reads their notes 6 times (thinking passes), rewrites the answer sheet once (writing pass), and hands it in again. The teacher marks it again. This repeats up to 16 times. The student's brain is a single tiny network. Re-reading notes is updating z. Rewriting the answer is updating y. Each hand-back is a supervision step.

What the Ablation Reveals

The paper systematically turns off each design choice on Sudoku-Extreme to measure what actually matters:

Configuration | Accuracy | Eff. Depth | ParamsTRM (T=3, n=6) | 87.4% | 42 | 5Mw/ 1-step gradient | 56.5% | 42 | 5Mw/ separate networks | 82.4% | 42 | 10Mw/ 4 layers, n=3 | 79.5% | 48 | 10MNo EMA | 79.9% | 42 | 5MHRM baseline | 55.0% | 24 | 27M

Sudoku-Extreme ablation. Effective depth = T(n+1) × layers per supervision step.

Training through all recursion steps is the biggest win. HRM only trained through the last 2 of its 6 network calls per block, arguing the rest had converged to a stable point. TRM trains through all 7. That single change accounts for a 31-point accuracy jump (56.5% to 87.4%). The "stable point" assumption was wrong, and it was costing HRM most of its learning signal.

A smaller network generalizes better. Cutting from 4 layers to 2 layers improved accuracy while halving the parameter count. With only ~1,000 training examples, the bigger network memorizes instead of learning patterns. The recursion already gives the model plenty of effective depth (42 layers worth per supervision step, 672 across all 16 steps), so the extra layers just add room to overfit.

One network beats two. Merging HRM's two separate networks into one gained 5 accuracy points and halved parameters. The shared weights act as a natural regularizer.

EMA prevents training collapse. Keeping a smoothed copy of the model weights (Exponential Moving Average, decay 0.999) stops the sharp divergence that otherwise occurs on small datasets.

Removing self-attention helps on small grids, hurts on large ones. Swapping attention for a simple MLP boosted Sudoku (9×9) by 13 points, but dropped ARC-AGI (30×30) from 44.6% to 29.6%.

Practical Implications

Edge deployment. A 5–7M parameter model runs on mobile devices, embedded systems, or in-browser. No cloud inference required. Sudoku-Extreme experiments ran on a single L40S GPU in under 36 hours. The compute barrier is orders of magnitude lower than LLM training.

Structured decision problems. Constraint satisfaction, combinatorial optimization, scheduling, anomaly detection: any problem where the answer is verifiable and iterative refinement makes sense is a candidate. This approach won't replace LLMs for open-ended generation, but it may outperform them on structured reasoning within resource-constrained environments.

Hybrid architectures. TRM-style refinement modules could sit downstream of LLMs: the language model generates an initial candidate, and a recursive refinement network iterates it toward correctness. This pipeline hasn't been tested yet, but the architectural compatibility is clear.

Caveats Worth Stating

  • Not a general-purpose model. TRM is supervised and task-specific. It produces one deterministic output, cannot explain its reasoning, and does not transfer across domains. Comparing it to LLMs on parameter count alone is apples-to-oranges.
  • The "small data" claim needs an asterisk. Each of the ~1,000 training examples is augmented 1,000 ways (shuffling, rotations, color permutations), making the effective training set closer to a million samples.
  • No theory for why recursion regularizes. The paper hypothesizes it relates to overfitting, but provides no formal analysis. A treatment through implicit regularization or information bottleneck theory would strengthen the claims.

Bottom Line

TRM demonstrates that for structured reasoning under data scarcity, deep recursion through a tiny shared-weight network generalizes better than both shallow passes through large networks and the more complex hierarchical approach of HRM. The single most important design decision is full backpropagation through all recursion steps; everything else is secondary.

The broader implication is that the relationship between model size, compute, and generalization is more nuanced than current scaling laws suggest. When the problem is structured and the data is limited, less really can be more.

References

  1. Jolicoeur-Martineau, A. (2025). "Less is More: Recursive Reasoning with Tiny Networks." arXiv:2510.04871.
  2. Wang, G., Li, J., Sun, Y., et al. (2025). "Hierarchical Reasoning Model." arXiv:2506.21734.
  3. Chollet, F. (2019). "On the Measure of Intelligence." arXiv:1911.01547.
  4. Chollet, F., Knoop, M., et al. (2025). "ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systems." arXiv:2505.11831.
  5. ARC Prize Foundation. (2025). "The Hidden Drivers of HRM's Performance on ARC-AGI." arcprize.org/blog/hrm-analysis.
  6. Wei, J., Wang, X., et al. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS 35.
  7. Snell, C., Lee, J., et al. (2024). "Scaling LLM Test-Time Compute." arXiv:2408.03314.
  8. Kaplan, J., McCandlish, S., et al. (2020). "Scaling Laws for Neural Language Models." arXiv:2001.08361.
  9. Tolstikhin, I. O., et al. (2021). "MLP-Mixer: An All-MLP Architecture for Vision." NeurIPS 34.
  10. Bai, S., Kolter, J. Z., & Koltun, V. (2019). "Deep Equilibrium Models." NeurIPS 32.
  11. Bai, X. & Melas-Kyriazi, L. (2024). "Fixed Point Diffusion Models." CVPR, 9430–9440.