Same Show
Every Night

Congratulations. You have just cast the most talented performer in showbiz. This actor can perform Shakespeare, explain tax law, write Python, and do a surprisingly good pirate accent — sometimes all in the same sentence. That is exactly the problem. The role you have cast them in is "Enterprise Backend Service, a drama in JSON," and the audience (your users, your auditors, your on-call engineer) expects the same performance every single night.

If you ask a normal function "what is 2+2" a thousand times, you get "4" a thousand times. If you ask your new star the same question, you might get "4", "Four!", "Great question — the answer is 4 🎉", and once, memorably, a haiku about arithmetic. The talent is real. The consistency is still a work in progress.

This is the basic comedy of building software on top of LLMs: we are casting improvisers in roles that were written for machines. The real answer — and the entire discipline of AI engineering with it — is that you do not fix this by asking the actor to behave better. You fix it by building a theatre around them.

ACT I — MEET THE TALENTWhy the actor never gives the same performance twice

An LLM does not "look up" an answer. Instead, it rolls weighted dice to choose the next word, again and again, at very high speed. If you turn the temperature setting up, the dice become looser and more random. If you turn it down to 0, you get what is called greedy decoding, which is mostly repeatable, but is still not a strict guarantee. Floating-point rounding, batching effects, and model updates on the provider's side mean that even your most predictable output can shift slightly between runs.

So let us set aside one idea early on: you will not make the actor deterministic. Improvisation is not a flaw in this performer. It is where the talent comes from. Determinism has to come from the system that surrounds the actor, not from the actor itself. This is the core idea of this article:

ACT II — HAND THEM A SCRIPTStructured outputs: freedom inside a fixed format

The first crew member you hire is the scriptwriter. Instead of asking the actor open-ended questions, such as "tell me about this invoice, in your own words," you hand them a script with fixed blanks to fill in. In practice, this means: your output is a JSON object, these are the fields, these are the types. You may improvise the content, but never the shape.

Every serious model API now supports this approach, through JSON schemas, structured outputs, and function or tool calling. It is the difference between an open-ended description of the customer's mood, and a clear, structured response like this:

{
  "sentiment":  "negative",          // enum: positive | neutral | negative
  "confidence": 0.87,                // number, 0–1
  "refund_eligible": true,           // boolean, no interpretive dance
  "reason_code": "DAMAGED_ITEM"      // enum, not freestyle poetry
}

Notice what happened here: the actor is still acting, using judgment and nuance to read between the lines of an angry customer email. But the output now arrives in a fixed format that your downstream code can check in a single line. Many complaints about "AI being unreliable" come down to asking for a vague, open-ended answer and then receiving one.

ACT III — THE SCRIPT SUPERVISORValidate, retry, and never fully trust a live performance

The next hire is the script supervisor — the person standing at the side of the stage with a clipboard, checking every line against the script. When the actor improvises something unplanned, the supervisor stops the take and says, "again, from the top, and this time the date field must be a proper date."

In code, this becomes a validation gate combined with a limited retry loop. First, parse the output against the schema. Then check the business rules that the schema alone cannot express: that the refund amount is not greater than the order total, or that the date is not something like the year 1847. If the check fails, retry the request with the error fed back to the model, since it is genuinely good at using that feedback. After a fixed number of attempts, fall back to a safe, deterministic path: a default value, a queue, or a human reviewer.

public async Task<RefundDecision> GetDecisionAsync(string email, Order order)
{
    for (var take = 1; take <= 3; take++)          // max three takes
    {
        var raw = await _model.CompleteAsync(_prompt.Render(email));

        if (!RefundDecision.TryParse(raw, out var d, out var errors))
        {
            _prompt.AddNote(errors);              // "again — with the fixes"
            continue;
        }
        if (d.Amount > order.Total)               // business rules: no method acting
        {
            _prompt.AddNote("refund exceeds order total");
            continue;
        }
        return d;                                 // a take we can print
    }
    return RefundDecision.EscalateToHuman();     // the understudy is a person
}

There is an important shift hiding inside that simple loop: the system's overall behaviour is now deterministic, even though the model's behaviour is not. Every possible outcome (a valid decision, a corrected decision, an escalation) was written by you, in advance, in ordinary code. The actor may surprise you, but the show itself cannot.

ACT IV — THE STAGE MANAGERThe deterministic sandwich

Now for the most important hire of all: the stage manager, the person who runs the show. The curtain rises at 8. Scene 2 always follows Scene 1. Props are set in advance. No matter how talented someone is, nobody decides in the middle of a performance to jump ahead to Act III.

In architectural terms: the LLM should be a small, clearly defined box inside a much larger deterministic system. Ordinary code decides when the model is called, what it is allowed to see, which tools it may use, and what happens with every possible shape of its answer. The overall workflow is a state machine that you wrote; the model fills in one creative slot at each state. This is what we call the "deterministic sandwich": plain code on top, plain code on the bottom, and improvisation only in the middle.

• Version your prompts the same way you version code. A prompt stored in a database field and edited casually is like an actor rewriting the script backstage without telling anyone. Instead, keep prompts in your code repository, review them through pull requests, and ship them with each release, so any change in behaviour can be traced back to a specific change.
• Cache your results aggressively. If the input, the prompt version, and the model version are all the same, serve the previously recorded result. Using idempotency keys on AI operations means a retry replays the same result, instead of generating a brand-new one.
• Pin your model versions. Using "latest" means the provider can effectively replace your lead actor overnight, without warning. Instead, pin the exact model version, upgrade only when you decide to, and re-run your evaluations (covered in the next section) before you go live with the new version.
• Give the model tools, not blind trust. Function calling with an allowlist is like giving an actor a prop with a fixed, safe handle: the actor may request an action such as "look up order status," but your deterministic code is what executes it, checks permissions, and decides what information comes back.

ACT V — DRESS REHEARSALS, FOREVEREvals: how you know the show still works

The final hire, and the one that most teams skip until a bad review forces their hand, is the rehearsal director. Traditional software tests ask a narrow question, like "does this function return 4?", which is not very useful for an actor who might correctly say "Four!" instead. Evaluations, or evals, ask a more realistic question: across 500 representative scenarios, how often is the performance acceptable, and did this week's change make it better or worse?

• Build a golden dataset made of real inputs with known, correct outcomes. This should be your standard set of test scenarios, including unusual ones, such as an angry email written in three languages, or a refund request for an item that was never ordered.
• Score results with plain code wherever possible — for example, checking schema validity, exact field values, or business-rule compliance. Where necessary, use a separate judge model for things like tone or faithfulness, but always have that judge grade against a clearly written rubric, not a vague impression.
• Gate every deployment on eval scores. A new prompt, a new model version, or a new temperature setting should never reach production without passing your evaluation numbers first. A prompt change that "reads better" but drops refund accuracy from 97% to 89% is a problem you want to catch during evaluation, not after it reaches your users.
• Keep evaluating even after you go live. Sample real, live traffic and feed it back into your evaluation pipeline. Drift is a real problem: your users change, the inputs change, and a system that performed well yesterday can perform worse today for weeks before anyone notices.

Evaluations are the answer to the question every stakeholder eventually asks, often with some suspicion: "but how do you KNOW it works?" The honest answer should never be "we tried it a few times and it seemed fine." It should be a specific number, shown on a chart, for every release.

Curtain call

So no, you never made the actor deterministic. Nobody ever has. What you built instead is a theatre around them: a script that fixes the output shape through schemas, a supervisor that checks every take through validation and limited retries, a stage manager that runs the show through deterministic orchestration, versioned prompts, pinned models, cached results, and allowlisted tools, and a rehearsal director who proves, with real numbers, that tonight's performance is at least as good as last night's, through evaluations.

Inside that theatre, the improviser is the star of the show. The talent stays, and the unpredictability stays behind a stage door it cannot wander out of. That is the whole craft of AI engineering, in one sentence:

Let the model improvise the lines. Never let it improvise the show. 🎭