From API wrapper to observable agent: how companies actually build their own AI

"Our company built its own AI" usually means something much smaller than it sounds. In most cases, it means the company put its logo on ChatGPT accessed through an API call. There is nothing wrong with that as a starting point, but it is worth understanding the full ladder, because each rung solves a real problem the previous one cannot, and each rung adds real complexity you should only pay for when you need it.

This is that ladder, from the thin wrapper to the observable agent, explained one rung at a time.

Level 0: The API wrapper

The wrapper is where nearly everyone starts. Your application takes user input, puts it inside a prompt, sends it to a model provider's API, and shows the response. A support chatbot, a "summarise this document" button, a writing assistant inside your product: most of these are wrappers.

Wrappers are fast to build, which is their virtue and their trap. They work beautifully in demos because demos use friendly inputs. In production they break in predictable ways: the model invents facts about your company, it forgets everything between requests, it cannot touch your data or your systems, and every answer costs you per token with no reuse.

None of this means wrappers are bad. It means they are a starting point with a ceiling, and you should know where the ceiling is before your users find it for you.

Level 1: The disciplined wrapper

Before reaching for new architecture, most teams should spend time at this level: the same API wrapper, but engineered instead of improvised.

The system prompt gets written deliberately, describing the assistant's role, its boundaries, and how it should behave when it does not know something. A few worked examples are included directly in the prompt, showing the model the shape of a good answer. Output is constrained to structured formats, usually JSON, so your application can parse it reliably instead of scraping prose.

This level also introduces the first operational concerns: retry logic for failed calls, timeouts, basic logging of what was asked and what came back, and cost tracking per request. None of this is glamorous. All of it is the difference between a demo and a feature.

Many products never need to go further than this. If your AI feature answers questions from general knowledge and takes no actions, a disciplined wrapper is the correct architecture, and anything fancier is resume-driven development.

Level 2: RAG, or giving the model your data

The moment your AI needs to know things the model was never trained on, your company's docs, your product catalogue, your support history, you need retrieval-augmented generation, universally known as RAG.

The idea is simple. You split your documents into chunks, convert each chunk into an embedding (a numeric representation of its meaning), and store those in a vector database. When a user asks a question, you search for the chunks most similar to the question, paste them into the prompt alongside it, and let the model answer from that material.

RAG solves the biggest wrapper problem: hallucination about your own domain. The model is no longer guessing what your refund policy says; it is reading it. But RAG introduces its own craft. Chunking decisions matter: too large and the retrieved context is noisy, too small and it loses meaning. Retrieval quality matters more than model choice; a great model with bad retrieval still makes things up. And someone has to keep the underlying documents current, because RAG over stale docs is confidently wrong at scale.

This is the level where "our own AI" starts to mean something. The model is still someone else's, but the knowledge is yours.

Level 3: The tool-using agent

Everything so far has been about answering. Agents are about doing. A tool-using agent is a model that can take actions: search the web, query your database, call your internal APIs, create tickets, send messages. The technique behind it is function calling: you describe your tools to the model in a structured way, and it decides, step by step, which tools to invoke and with what arguments.

This is a genuine phase change. A wrapper answers the question "what is our refund policy." An agent handles "refund this customer's last order": it looks up the order, checks the policy, issues the refund through your payments API, and tells the customer it is done.

The complexity jumps accordingly. An agent makes sequences of decisions, and each decision can go wrong in ways that compound. It needs guardrails: which tools it may call unsupervised, which need human approval, what it must never do. It needs idempotency, because it will occasionally call the same tool twice. And it needs a way to recover when a step fails halfway through a plan, which is the normal case, not the edge case.

This is also where costs stop being predictable. A wrapper costs roughly one model call per user request. An agent can make dozens of calls, plus tool executions, for a single task. You need budgets per task and circuit breakers, or one confused agent will spend your infrastructure budget exploring.

Level 4: The observable agent

Here is the rung almost nobody talks about and everybody needs: observability. An agent you cannot observe is a liability wearing a demo.

Observable means you can answer five questions about every agent run. What did it do, step by step, in a trace you can read. Why did it do it, with the model's reasoning visible at each decision. What it cost, in tokens and tool calls, attributed to the task. Whether it was right, measured against evals you wrote, not vibes. And where it failed, with enough context to fix the cause instead of the symptom.

In practice this means structured logging of every model call and tool invocation, trace visualisation so a human can replay a run, evaluation suites that run your agent against known scenarios on every change, and dashboards tracking success rates, costs, and failure modes over time. Tools exist for this layer, Langfuse and LangSmith are the names you will hear most, but the tooling matters less than the habit: no agent ships without traces, no change ships without evals.

This is what separates the companies with AI in production from the companies with AI in a slide deck. The model is the easy part. Knowing what it did is the job.

The honest map

Most teams should climb exactly as far as their problem demands and stop. A support FAQ bot is a level 1 problem. A documentation assistant over private docs is level 2. Automating multi-step operational work is level 3, and it earns level 4 the day it touches anything real.

The pattern to resist is starting at level 3 because agents are exciting. Every rung you skip is a failure mode you will discover in production, billed by the token, explained to you by an angry customer. The ladder exists so you can climb it deliberately, not so you can jump.