Ask what makes an AI agent different from a chatbot and you will usually be told it "can take actions" or "works autonomously." Both are true and neither explains anything. The model at the centre of an agent is the same model that powers the chatbot, and it is capable of exactly one thing: text in, text out. It cannot read a file. It cannot call an API. It cannot spend your money.
What makes it an agent is a loop — and the loop is ordinary software that you or your framework wrote. The model emits a request to use a tool. Your code reads that request, performs the action, writes the result back into the conversation, and asks the model again. Round and round until the model stops asking.
Understanding that one structural fact explains almost everything else that puzzles people about agents: why they can run away with your budget, why they sometimes stop halfway through and claim they're finished, why "give it more tools" often makes them worse, and why the bill grows faster than the transcript. Here is the mechanism.
The Model Doesn't Do Anything
Start with the part that gets skipped.
When an agent "searches the web," the model does not search the web. It produces a structured block of text that amounts to: I would like to call the tool named web_search with the query nvidia q3 earnings. That block is the end of the model's turn. Nothing has happened yet.
Your program receives it, and your program decides what to do. It might run the search. It might refuse. It might ask the user for permission first. It might return an error. Whatever it does, it then appends the outcome to the conversation as a new message and sends the whole thing back to the model.
This is why the phrase "the agent deleted my files" is, strictly speaking, never accurate. The model asked. Something on your side said yes. That distinction is not pedantry — it is where every meaningful safety control lives, because it is the only point in the system where a human or a rule can intervene.
One Turn, Step by Step
Formally the cycle has four steps:
- Send. Your code posts the conversation so far plus the list of available tools.
- Receive. The model returns either a final answer or one or more tool requests.
- Execute. Your code runs whatever was requested and captures the result — including failures.
- Append. The result goes into the conversation as a new message, and you go back to step 1.
The loop exits when step 2 produces plain text instead of a tool request. That's it. There is no separate "done" signal, no completion callback, no internal sense of having finished a job.
The intellectual ancestor of this pattern is a 2022 paper, ReAct: Synergizing Reasoning and Acting in Language Models by Yao and colleagues, which showed that interleaving reasoning traces with actions beat doing either alone. Reasoning let the model plan and recover from errors; acting let it fetch facts it didn't have. On two interactive benchmarks, ALFWorld and WebShop, the approach beat imitation-learning and reinforcement-learning baselines by 34 and 10 absolute percentage points of success rate — with only one or two examples in the prompt. Nearly every agent framework in production today is a descendant of that loop.
Why the Whole Conversation Is Re-Sent Every Single Turn
Here is the fact that surprises people most: the model has no memory of the previous turn.
The API is stateless. It does not store your conversation. On turn 12 of an agent run, your code sends turns 1 through 11 again, in full, along with the system prompt and every tool definition. The illusion of continuity is manufactured entirely on your side, by resending everything.
That has a direct consequence for cost, and it is arithmetic rather than opinion.
Take an agent with a 2,000-token setup — system prompt plus tool definitions — where each turn adds roughly 500 tokens of model output and tool results. Turn 1 bills 2,000 input tokens. Turn 2 bills 2,500. Turn 20 bills 11,500. Add up all twenty turns and you have been billed 135,000 input tokens to move a conversation that ends up 12,000 tokens long — 11.25 times the content.
A linear conversation produces a quadratic bill. Nothing is malfunctioning; this is what statelessness costs.
The mitigation is prompt caching: the provider stores the processed form of a stable prefix and charges a fraction of the normal rate to read it back. Anthropic's documented pricing puts cache reads at roughly a tenth of the base input price, which changes the economics of a long run considerably. But caching is a prefix match, and that has a design consequence most teams learn the hard way: interpolate a timestamp or a session ID near the top of your system prompt and every byte after it is uncacheable. Put the stable content first and the volatile content last.
Six Ways a Turn Can End
This is where real agents break, and it is almost always the same bug.
Each response carries a stop reason saying why the model stopped generating. Most tutorials cover two of them and stop there. In Anthropic's API there are six, and they are not interchangeable.
| Stop reason | What it means | What your loop should do |
|---|---|---|
tool_use | The model wants a tool run | Execute, append the result, continue |
end_turn | The model considers itself finished | Exit the loop |
pause_turn | A server-side tool loop hit its own iteration limit | Re-send the conversation to resume |
max_tokens | The answer was cut off at the output cap | Don't treat this as an answer — raise the cap or stream |
stop_sequence | A stop string you configured was matched | Exit, and check whether that was intended |
refusal | Safety classifiers declined the request | Check this before reading the content, which may be empty |
Two of these deserve a closer look.
pause_turn is a distinct kind of loop. When the model uses a server-side tool — a hosted web search, say — the provider runs its own internal loop, and that loop has a limit of its own. Anthropic's documentation puts the default at 10 iterations, after which the response comes back marked pause_turn. It is not an error and not a completion; it means "I'm mid-task, ask again." Code that treats it as end_turn will report a half-finished answer as final.
refusal breaks naive parsing outright. When safety classifiers decline a request, the content array may be empty. Code that reaches straight for the first content block — an overwhelmingly common pattern — crashes or silently returns nothing. Checking the stop reason before reading content costs one line and prevents a whole class of production incident.
The Failure Modes Nobody Warns You About
Runaway loops. Nothing in the mechanism guarantees termination. A model that keeps asking for tools keeps getting them, and each round costs money and time. Anthropic's own engineering guidance on building effective agents is explicit that "it's also common to include stopping conditions (such as a maximum number of iterations) to maintain control." An iteration cap is not a nicety; it is the only hard backstop in the design.
Parallel calls, mishandled. A single model turn can contain several tool requests at once — fetch three files, run two searches. The correct handling is to execute them and return every result inside a single message. Split them across separate messages and you are, in effect, training the model by example that parallel calls don't work; it will start serialising, and your agent gets slower for reasons that look inexplicable from the outside.
Swallowed errors. When a tool fails, the temptation is to drop the result and retry quietly. Don't. Pass the failure back, flagged as an error. The model is often good at adapting — trying a different path, or telling the user what's blocked. An agent that never sees its own failures cannot route around them.
Context exhaustion. Long runs fill the context window with old tool output that no longer matters. There are three distinct remedies and they are not the same thing: pruning (drop stale tool results), summarising (compact the history into a shorter form), and external memory (write findings to a file the agent can re-read later). Agents that run for hours generally need all three.
Silent no-ops. The strangest failure: the model writes what looks like a tool call into its ordinary text output instead of emitting a proper tool request. The turn completes successfully, no error is raised — and the tool never runs. From the outside it looks like the agent did the work and lied about it.
The Question to Ask First: Do You Need a Loop at All?
The most useful distinction in this whole field is between a workflow and an agent. Anthropic's engineering team draws it cleanly: workflows are "systems where LLMs and tools are orchestrated through predefined code paths," while agents are "systems where LLMs dynamically direct their own processes and tool usage."
If you already know the steps — extract the invoice fields, look up the vendor, write the row — that is a workflow. Write it as ordinary code with model calls inside it. It will be cheaper, faster, more debuggable, and it cannot loop forever.
The loop is worth its cost only when the path genuinely can't be specified up front: an open-ended investigation, a bug of unknown origin, a research question whose next step depends on what the last step returned. The same guidance is blunt about the trade: "agentic systems often trade latency and cost for better task performance, and you should consider when this tradeoff makes sense."
| Workflow | Agent | |
|---|---|---|
| Who decides the next step | Your code | The model |
| Number of model calls | Known in advance | Unknown until it stops |
| Cost predictability | Bounded | Bounded only by your iteration cap |
| When to prefer it | The steps are knowable | The path depends on what you find |
If you're deciding between these for the first time, our practical guide to AI agents covers the decision in more detail, and the AI agents topic hub collects the rest of our coverage. When a single loop stops being enough and work needs to fan out, that becomes a multi-agent system — several loops sharing a workspace, with its own set of failure modes.
Where Tool Quality Actually Comes From
One counterintuitive finding: the highest-leverage thing you can improve in an agent is usually not the prompt, the model, or the loop. It's the tool descriptions.
The model chooses tools from their descriptions alone. A vague one-liner produces wrong choices no amount of prompt engineering can fix, and the common failure is under-description rather than over-description. A good description says what the tool does, when to use it, when not to, what each parameter means, and what the tool does not return. Anthropic's guidance treats this interface — what it calls the agent-computer interface — as deserving the same care as a user-facing API, with "thorough tool documentation and testing."
Two related points fall out of the same idea. Fewer, clearly bounded tools beat many overlapping ones, because overlap forces a judgement call the model has no basis to make. And retrieval is just another tool: an agent that can search a document store is running RAG inside the loop, and the same description quality decides whether it reaches for it at the right moment.
Common Misconceptions
- "The agent runs on the model's side." It doesn't. The loop runs on yours. The model is a stateless function you call repeatedly.
- "Agents remember previous turns." They don't. Your code resends the history. What looks like memory is retransmission.
- "More tools make an agent more capable." Past a point they make it worse, because tool selection becomes ambiguous.
- "An agent knows when it's finished." It emits a stop reason. Whether that means finished is your code's interpretation — and there are six possibilities, not two.
- "Autonomy is a model property." Autonomy is a property of the loop and its permissions. Removing the iteration cap doesn't make a model smarter; it just removes your brakes.
The Bottom Line
An agent is a while loop. The model proposes; your code disposes; the result goes back into the conversation; repeat until the model stops proposing. Everything that feels magical about agents, and everything that goes wrong with them, follows from that shape.
Which means the engineering work is mostly not model work. It's writing tool descriptions good enough to choose from, handling all six stop conditions rather than the obvious two, capping iterations so a bad run ends, passing errors back instead of hiding them, and managing a context window that fills up with yesterday's tool output. None of that is glamorous, and all of it is what separates an agent that works from a demo that impressed someone once.
The best question to ask before building one is still whether you need the loop at all. If you can write down the steps, write down the steps.
Frequently Asked Questions
Is an agent a different kind of model?
No. It's the same model, called repeatedly inside a loop. Some models are trained to be better at tool use and at staying coherent over long runs, but there is no separate "agent model" architecture. What you're buying when you buy an agent product is the loop, the tool integrations, the context management and the safety controls around a model — not a different model.
Who actually executes the actions?
Your code, or the framework you're using. The model returns a structured request naming a tool and its arguments; something on your side reads that request and decides whether to carry it out. This is why permission prompts and approval gates are possible at all — the gap between the request and the execution is where they live.
Why do agents get more expensive the longer they run?
Because the API is stateless, so every turn resends the entire conversation. A run whose transcript grows linearly produces a bill that grows quadratically. In a worked example of a 2,000-token setup growing 500 tokens per turn, twenty turns bill 135,000 input tokens to carry 12,000 tokens of content. Prompt caching is the standard mitigation, repricing the repeated prefix at a fraction of the normal input rate.
What stops an agent from looping forever?
Only a limit you impose. The mechanism has no natural termination — a model that keeps requesting tools keeps being served. Standard practice is a maximum iteration count, often combined with a token or spend budget. Some providers also expose an advisory budget the model itself can see, so it paces its work and wraps up rather than being cut off mid-sentence.
Why does an agent sometimes stop and claim it's done when it isn't?
Usually a mishandled stop reason. If a turn ends at the output-token cap, or a server-side tool loop pauses, or safety classifiers decline the request, the loop can read any of those as completion if it only checks for the two obvious values. The rarer version is the model writing a tool call into its visible text rather than emitting a real request — the turn succeeds, the tool never runs, and nothing reports an error.
Do I need an agent, or would a workflow do?
If you can write the steps down in advance, write them down — that's a workflow, and it will be cheaper, faster and easier to debug. Reach for a loop only when the path genuinely can't be specified up front, such as an open-ended investigation where each step depends on what the previous one returned. Agentic systems trade cost and latency for the ability to handle tasks you couldn't script.
What's the single biggest lever on agent quality?
Tool descriptions, more often than the prompt or the model. The model picks tools from their descriptions alone, so a vague description produces wrong choices that better prompting cannot fix. Say what the tool does, when to use it, when not to, what every parameter means, and what it does not return — and prefer a few clearly bounded tools over many overlapping ones.
Sources
- ReAct: Synergizing Reasoning and Acting in Language Models — Yao et al., arXiv:2210.03629 (2022), ICLR 2023
- Building effective agents — Anthropic engineering
- Tool use overview — Anthropic developer documentation
- Handling stop reasons — Anthropic developer documentation
- Prompt caching — Anthropic developer documentation
