Running an LLM in production is mostly about everything around the model: a test set that tells you when it gets worse, tools with narrow permissions, a budget for every run, a log of every step, and a clear way back to a person. The prototype proves the model can do the task once. Production has to prove it does the task every day, at a known cost, without doing anything you did not allow. This note covers what I add between those two points.
LLM in production: the short answer
An agent that works in a demo usually lacks five things. Before it goes live, I want each of them in place:
- an eval set: real inputs with expected outputs, run on every change;
- tools that do one thing each, with the least privilege they need;
- limits per run: steps, time, tokens and money;
- a trace of every model call and tool call, with inputs, outputs and cost;
- a fallback: what happens when the agent is unsure, fails, or hits a limit.
All five work the same whichever model you pick: they are ordinary engineering applied to a component that is non-deterministic.
Workflow or agent: decide that first
Many projects called "agents" work better as workflows. Anthropic draws the line clearly in Building effective agents: "Workflows are systems where LLMs and tools are orchestrated through predefined code paths", while agents "dynamically direct their own processes and tool usage". The same guide recommends "finding the simplest solution possible, and only increasing complexity when needed", and warns that "the autonomous nature of agents means higher costs, and the potential for compounding errors."
My rule: if I can write the steps down in advance, the code decides the order and the model handles the fuzzy parts (reading a document, classifying a request, drafting a reply). I let the model choose the steps only when the path really changes from case to case. A workflow is easier to test, cheaper to run and easier to explain when something goes wrong.
An eval set before the code
The eval set is the contract. It is a list of real inputs, collected with the people who do the work today, each with the expected result or the rules a good result must follow. I build it before writing the agent, and I run it on every change to the prompt, the tools or the model.
What it catches:
- regressions when you change a prompt to fix one case and break three others;
- differences between models, so you choose on your own data and not on a leaderboard;
- drift after a provider updates a model behind the same name.
Some checks are exact (a field must contain a valid invoice number). Others need a judgment, and for those I use a rubric that a second model or a person scores. After launch, every real failure becomes a new case in the set.
Tools: few, narrow, well documented
An agent is only as safe as the tools it can call. The OWASP Top 10 for LLM Applications lists this risk as LLM06:2025 Excessive Agency, with three root causes: "excessive functionality", "excessive permissions" and "excessive autonomy". Its first recommendation is to "limit the extensions that LLM agents are allowed to call to only the minimum necessary."
In practice:
- a tool that reads customer records does not also update them;
- each tool runs with its own credentials, scoped to what it does;
- tools that change data, spend money or send messages ask for confirmation;
- write operations are idempotent, so a retry never does the work twice.
The tool descriptions matter as much as the code. Anthropic's guide says "it is therefore crucial to design toolsets and their documentation clearly and thoughtfully". The model only knows what the description tells it.
If the agent is exposed through MCP, the protocol's security best practices are worth reading in full. One rule I apply everywhere: "MCP servers MUST NOT accept any tokens that were not explicitly issued for the MCP server."
Cost and limits per run
Every run needs a ceiling. OWASP's LLM10:2025 Unbounded Consumption recommends rate limiting, user quotas, and to "set timeouts and throttle processing for resource-intensive operations". Anthropic suggests stopping conditions "such as a maximum number of iterations".
What I set on every agent:
| Limit | Why |
|---|---|
| Maximum steps per run | Stops loops where the agent retries the same failing call |
| Timeout per run and per tool | A slow dependency should not hold the run open |
| Token and money budget per run | A long input or a bad loop cannot burn the monthly budget |
| Quota per user or workspace | One heavy user cannot starve the others |
I also record the real cost of each run. Without that number you cannot price the feature, and you find out about a problem from the invoice. For what the build itself costs, I wrote a separate page on AI agent development cost.
Observability: log every step
When an agent gives a wrong answer, you need to see why. That means a trace for every run: the input, each model call with its prompt and response, each tool call with arguments and result, timings, token counts and cost. With that trace you can tell whether the model misread the task, a tool returned bad data, or the retrieval step missed the right document.
The trace also feeds the eval set. A complaint from a user becomes a reproducible case in minutes. Keep personal data out of the logs or mask it, and decide how long traces are retained.
Untrusted input and prompt injection
Anything the agent reads from outside can carry instructions. OWASP describes indirect prompt injection as what happens when an LLM "accepts input from external sources, such as websites or files", and notes that techniques like RAG and fine-tuning "do not fully mitigate prompt injection vulnerabilities."
I treat this as a permissions problem. Content from web pages, emails and uploaded files is data, and the agent's ability to act on it is limited by the tools it has. If an injected instruction says "send this file to an outside address" and the agent has no tool that sends files outside, the attack has nowhere to go. Retrieval brings its own risks here, which I cover in the note on enterprise RAG.
The human fallback
A production agent needs a defined way out. When confidence is low, a limit is reached or a tool fails twice, the run stops in a known state and a person gets the case with the trace attached. For actions that are hard to undo, a person approves before they happen. Anthropic's guide recommends "extensive testing in sandboxed environments, along with the appropriate guardrails"; the fallback is the guardrail that matters most on a bad day.
What it looked like in a real system
My most complete example is TreZeroUno, a platform I built that exposes more than 150 tools to Claude through an MCP server. What it took to make that safe to use every day:
- every paid tool states its price first and runs only within a spending limit the user approved;
- requests carry an idempotency key, so repeating one never charges twice;
- long tasks run as jobs in a queue, with progress, cancellation and refunds for the part not done;
- spending caps per workspace and a ledger of real costs for every provider;
- content from websites and AI answers is handled as untrusted data, never as instructions.
Every item on that list is about running the model safely. If you are planning an agent of your own, my page on AI agent development explains how I take one from prototype to production.
Frequently asked questions
What are LLM agents?
An LLM agent is a system where a large language model decides which steps to take and which tools to call to reach a goal, using the result of each step to choose the next. The model is one component; the agent also includes the tools, permissions, memory, limits and checks around it.
Is ChatGPT an LLM or an agent?
ChatGPT is an application built on large language models. The model on its own generates text. When the application lets it browse, run code or call other tools and decide the next step, that part behaves like an agent.
How do you deploy an LLM in production?
Start from a narrow task and an eval set built from real cases. Wrap the model in code that controls tools, permissions and limits, log every step, and ship it to a small group first with a human fallback. Widen access only when the eval results and the traces from real use hold up.
What does it cost to run an AI agent?
It depends on the number of model calls per run, the size of the inputs and the model chosen for each step. Measure the real cost per run on your eval set before launch, then set a budget per run and per user so the total stays predictable.