A base model is a horse. Fast, strong, and perfectly happy to run in the wrong direction. Give it a task that spans more than one turn and it forgets what it did three steps ago, repeats work it already finished, or keeps calling tools until something breaks or the bill does.

The fix isn't a bigger model. It's the harness around it: a memory layer that gives the model a past, a retrieval layer that decides what to pull when, and a control loop that decides when to stop. Most of what people call "building an agent" is building this scaffolding. The prompt is maybe a tenth of the work.

Memory is three problems, not one

Lumping memory into a single vector store is the first mistake. Cognitive science splits memory into kinds that behave differently, and agents need the same split because the storage and retrieval mechanics don't match across them.

Procedural: how to do things

Skills and behavior. How the agent formats a report, which validation runs before a write, the house style for a commit message. This is runbook material, and the cheapest place to keep it is plain Markdown the model reads at load time. No embedding step, no similarity query. It's just in context. Fast, legible, diffable in git. When a skill file is wrong you can open it and fix it, which is not true of a weight or a buried embedding.

Semantic: what's true

Durable facts and distilled knowledge. Product docs, past architectural decisions, the schema of the database the agent queries. This is the RAG case people actually mean when they say RAG: embed it, store the vectors, pull top-k by similarity at query time. The failure mode is staleness. Semantic memory that nobody re-indexes slowly starts lying to you.

Episodic: what happened

The log of events. What the user asked yesterday, which tool call failed overnight, the exact sequence that produced the current state. This is time-series and relational, not semantic. "What did I do in the last hour" is a WHERE timestamp > ... query, not a cosine-similarity search. Forcing episodic recall through a vector store is a common and expensive error: you get fuzzy near-matches when what you needed was an ordered, exact history.

Retrieval is routing, not lookup

Once memory is three stores, retrieval stops being one operation. It becomes routing: work out what kind of question you're answering, then hit the store that can answer it. Semantic questions go to the vector index. Historical questions go to the episodic tables. A lot of agent quality lives in this unglamorous decision, and getting it wrong is exactly why an agent "forgets" something it technically holds. It looked in the wrong place.

Route badly and you either over-retrieve, stuffing the context with similar-but-irrelevant chunks, or under-retrieve and miss the one row that mattered. Neither throws an error. Both show up as an agent that seems dumb for no visible reason.

The loop is where agents actually break

Memory gives the model a past. The loop runs it forward: call the model, take an action, observe the result, repeat. The repeat is the dangerous part, because nothing in a base model wants to stop.

Stopping conditions and guardrails

An agent with tool access and no stopping condition is a while-loop with a credit card. You need explicit exits: a max-step budget, a confirmation gate before anything irreversible like sending the email or running the migration, a check that the last action actually changed state. When people say an agent "acted blindly," it's rarely the model being stupid. It's a guard the engineer didn't write.

Tracing, and the tree-of-thought mixup

When a ten-step run goes wrong, "the agent gave a bad answer" tells you nothing. You need the trace: every model call, every tool input and output, the branch taken at each decision point. This is observability, the LangSmith-and-OpenTelemetry layer, and it's the gap between debugging in an hour and staring at a black box.

One thing worth pulling apart, because the two keep getting drawn as the same box: tracing and Tree of Thought are not the same thing. Tracing is how you watch an agent's execution after it runs. Tree of Thought is a reasoning method, where you have the model branch into several candidate paths and score them before committing to one. Both are trees on a whiteboard. One is an ops tool, one is a prompting strategy. Treat them as interchangeable and you'll go hunting for Tree of Thought in your monitoring dashboard.

Evals close the outer loop

The outermost loop is the ops one. Run the agent against a fixed set of cases, score the outputs, change one thing (the prompt, the model, a retrieval parameter), then measure whether the score moved. Skip this and you're not engineering, you're vibing. The scoring doesn't need to be clever. Thirty hand-labeled cases with a pass/fail check beat "it felt better in the demo" every time.

The model was always going to be fast. Whether it runs the race or bolts into traffic comes down entirely to what you strapped on before you let go.