Same Show
Every Night

Congratulations. You have just cast the most talented performer in the business. This actor can do Shakespeare, explain income tax, write Python and put on a surprisingly good pirate accent, sometimes in the same sentence. And that is exactly the problem. The role you gave them is "Enterprise Backend Service, a drama in JSON", and the audience (your users, your auditors, your on-call engineer) expects the same performance every single night.

Ask a normal function "what is 2+2" a thousand times and you get "4" a thousand times. Ask your new star the same question and you might get "4", "Four!", "Great question — the answer is 4 🎉", and once, unforgettably, a haiku about arithmetic. The talent is real. The consistency is still being worked on.

This is the basic comedy of building software on top of LLMs: we are casting improvisers in roles written for machines. The real answer, and the whole discipline of AI engineering along with it, is that you do not fix this by asking the actor to behave. You fix it by building a theatre around them.

ACT I — MEET THE TALENTWhy the actor never gives the same performance twice

An LLM does not "look up" an answer. It rolls weighted dice to choose the next word, again and again, very fast. Turn the temperature setting up and the dice become more random. Turn it down to 0 and the model picks the most likely word each time, which is mostly repeatable but still not a promise. Tiny rounding differences, how the provider groups requests together, and model updates on their side mean even your most predictable answer can change slightly from one run to the next.

So let us drop one idea right at the start: you will not make the actor give the same answer every time. Improvising is not a defect in this performer. It is where the talent comes from. The consistency has to come from the system around the actor, not from the actor. That is the main idea of this article:

ACT II — HAND THEM A SCRIPTStructured output: freedom inside a fixed format

The first crew member you hire is the scriptwriter. Instead of asking the actor an open question like "tell me about this invoice in your own words", you give them a script with fixed blanks to fill. In practice this means: your answer is a JSON object, these are the fields, these are their types. You can improvise what goes inside, but never the shape.

Every serious model API supports this now, through JSON schemas, structured outputs and function (tool) calling. It is the difference between a long paragraph about the customer's mood and a clear, structured answer like this:

{
  "sentiment":  "negative",          // enum: positive | neutral | negative
  "confidence": 0.87,                // number, 0–1
  "refund_eligible": true,           // boolean, no interpretive dance
  "reason_code": "DAMAGED_ITEM"      // enum, not freestyle poetry
}

See what happened here. The actor is still acting, using judgement to read between the lines of an angry customer email. But the answer now comes in a fixed format that your code can check in one line. Many complaints that "AI is unreliable" come from asking for a vague, open answer and then getting exactly that.

ACT III — THE SCRIPT SUPERVISORCheck, retry, and never fully trust a live performance

The next hire is the script supervisor, the person at the side of the stage with a clipboard, checking every line against the script. When the actor goes off-script, the supervisor stops the take and says, "Again, from the top, and this time the date field must be a real date."

In code, this is a validation check plus a retry loop with a limit. First, check the answer against the schema. Then check the business rules the schema cannot express: the refund is not more than the order total, and the date is not something like the year 1847. If a check fails, try again and send the error back to the model, because models are good at fixing a mistake once you show it to them. After a fixed number of tries, fall back to a safe path you control: a default value, a queue, or a human reviewer.

public async Task<RefundDecision> GetDecisionAsync(string email, Order order)
{
    for (var take = 1; take <= 3; take++)          // max three takes
    {
        var raw = await _model.CompleteAsync(_prompt.Render(email));

        if (!RefundDecision.TryParse(raw, out var d, out var errors))
        {
            _prompt.AddNote(errors);              // "again — with the fixes"
            continue;
        }
        if (d.Amount > order.Total)               // business rules: no method acting
        {
            _prompt.AddNote("refund exceeds order total");
            continue;
        }
        return d;                                 // a take we can print
    }
    return RefundDecision.EscalateToHuman();     // the understudy is a person
}

That simple loop hides a big change: the system as a whole now behaves the same way every time, even though the model does not. Every possible outcome (a valid answer, a corrected answer, or a hand-over to a person) was written by you, in advance, in normal code. The actor can surprise you. The show cannot.

ACT IV — THE STAGE MANAGERThe predictable sandwich

Now the most important hire of all: the stage manager, the person who runs the show. The curtain goes up at 8. Scene 2 always comes after Scene 1. Props are placed in advance. However talented someone is, nobody decides in the middle of the show to jump to Act III.

In architecture terms: the LLM should be one small, clearly defined box inside a much bigger system that you control. Normal code decides when the model is called, what it can see, which tools it can use, and what happens for every possible type of answer. The overall flow is a set of steps you wrote, and the model fills one creative slot at each step. Call it a sandwich: plain code on top, plain code at the bottom, and improvisation only in the middle.

• Version your prompts just like code. A prompt kept in a database field and edited casually is like an actor rewriting the script backstage without telling anyone. Keep prompts in your code repository, review them in pull requests and ship them with each release, so every change in behaviour can be traced to a specific change.
• Cache results wherever you can. If the input, the prompt version and the model version are all the same, return the answer you already have. Give each AI operation a unique request key, so a retry returns the same saved result instead of creating a fresh one.
• Fix the model version. Using "latest" means the provider can swap your lead actor overnight without telling you. Pin the exact model version, upgrade only when you decide to, and rerun your evaluations (next section) before the new version goes live.
• Give the model tools, not blind trust. Function calling with an approved list is like giving an actor a prop with a safe, fixed handle. The actor can ask for an action like "check order status", but your own code carries it out, checks permissions and decides what information goes back.

ACT V — DRESS REHEARSALS, FOREVEREvals: how you know the show still works

The last hire, and the one most teams skip until a bad review forces them, is the rehearsal director. Normal software tests ask a narrow question like "does this function return 4?", which does not help much with an actor who might rightly say "Four!" instead. Evaluations, or evals, ask a more practical question: across 500 typical scenarios, how often is the performance acceptable, and did this week's change make it better or worse?

• Build a golden dataset of real inputs with known correct answers. This is your standard set of test scenarios, including the odd ones, like an angry email written in three languages, or a refund request for something that was never ordered.
• Score results with plain code wherever you can, for example checking the schema, exact field values or business rules. Where you have to, use a separate judge model for things like tone or accuracy, but always make that judge mark against a clearly written rubric, not a vague feeling.
• Block every release that fails the evals. A new prompt, a new model version or a new temperature setting should never reach production without passing your evaluation numbers first. A prompt change that "reads better" but drops refund accuracy from 97% to 89% is something you want to catch in testing, not after users see it.
• Keep evaluating after you go live. Take a sample of real traffic and feed it back into your evaluations. Things drift: your users change, the inputs change, and a system that worked well yesterday can quietly get worse for weeks before anyone notices.

Evals answer the question every stakeholder asks sooner or later, often with some doubt: "But how do you KNOW it works?" The answer should never be "we tried it a few times and it looked fine". It should be a specific number, on a chart, for every release.

Curtain call

So no, you never made the actor predictable. Nobody ever has. What you built instead is a theatre around them: a script that fixes the shape of the answer (schemas), a supervisor who checks every take (validation and limited retries), a stage manager who runs the show (your own code controlling the steps, versioned prompts, fixed model versions, cached results and approved tools), and a rehearsal director who proves with real numbers that tonight's show is at least as good as last night's (evals).

Inside that theatre, the improviser is the star. The talent stays, and the unpredictability stays behind a stage door it cannot walk out of. That is the whole craft of AI engineering in one sentence:

Let the model improvise the lines. Never let it improvise the show. 🎭