White paper · For operators · October 2026
After the demo
Why AI pilots stall after a good demo, and the five things that get them into production.
The argument in brief
- Why good demos stallA demo shows an idea can work once, and production needs it to work every time, with nobody watching, at a cost you know in advance.
- Building for scaleWhat works once has to become software that runs the same way every time, on numbers everyone agrees on.
- What to measureDecide what success looks like before the build starts, in numbers the CFO already reports.
- Why evals come firstAn eval is a test for an AI system, and it should exist before the system does.
- What it costs to runA demo costs almost nothing, but production has running costs, and they should be budgeted per task.
- A roadmap for the first yearPlan the first year as one sequence with gates, starting in finance, instead of a string of pilots.
Introduction
Most AI pilots we come across start with a good demo. An agent answers questions or clears a queue on a dozen hand-picked examples, the room is impressed, and six months later the agent still isn’t part of how the work gets done.
The gap between a demo and production is rarely the model; in our work, it comes down to five things. They are building it to scale, agreeing what to measure, writing evals, budgeting for running costs and following a roadmap. This paper takes each in turn and follows our June paper, The company brain, which was about bringing a company’s knowledge into one place that AI can use safely.
Why good demos stall
A demo shows an idea can work once, and production needs it to work every time, with nobody watching, at a cost you know in advance.
A demo runs on a handful of clean examples, with the person who built it watching and nobody counting the cost. Production means thousands of messy cases, nobody watching most of them, a bill every month and someone accountable when an answer is wrong. Most of what breaks between the two has little to do with the AI.
The most common reason we see a pilot stall is that the operation underneath it isn’t ready. Three questions show it quickly: can someone write down the steps of the work, does every request arrive labeled and routed, and is anyone measuring how often the work goes wrong today? If the answer to any of them is no, there’s no standard to hold an agent to, and that work comes before the agent does.
Exhibit 1
What changes between a demo and production
Building for scale
In the demo
An agent does the task well once, with the person who built it watching
In production
What it did is saved as software with a test, and runs the same way every time
What to measure
In the demo
Judged on how the demo felt
In production
A baseline taken before anything changes, and four numbers tracked for every agent
Evals
In the demo
A dozen hand-picked examples
In production
Real cases with known right answers, and a pass mark agreed in advance
Running costs
In the demo
A few hours of someone’s time
In production
Model fees, review, upkeep and licenses, budgeted per completed task
Roadmap
In the demo
A string of pilots
In production
One sequence with gates, starting in finance
Building for scale
What works once has to become software that runs the same way every time, on numbers everyone agrees on.
An agent that does a task well once is a demo. To scale it, have the agent do the task once while a person checks the result, then save what it did as ordinary software with a test. From then on the software does the work, and the agent comes back only when the test fails.
The same applies to any rule the business already enforces by hand, such as a pricing policy, an approval limit or a documentation requirement. Written once as code, the rule can be checked on every case, and a person only reviews the exceptions.
Two other things decide whether one agent can become twenty. Every agent should read the same agreed definition of each number, and every agent should work under the same access rules as the people it works for. If someone can’t open the board deck, neither can the agent working for them, and every action an agent takes can be traced back to a person.
Exhibit 2
One supervised run, then software that runs on its own
- Step 1The agent does the task once
- Step 2A person checks the result
- Step 3What it did is saved as ordinary software, with a test
- Step 4From then on, the software does the work
What to measure
Decide what success looks like before the build starts, in numbers the CFO already reports.
Most pilots are judged on how the demo felt, but production needs a baseline taken before anything changes: how long the work takes today, what it costs and how often it goes wrong. Against that baseline, track four numbers for every agent: the cost of each completed task, the share of tasks finished without a person, accuracy on the eval set, and the time from request to result.
Each of those numbers needs one agreed definition, or the reports soon start to disagree. Keep each definition in one place that every report and agent uses, and add a check that flags any figure that differs between two places, such as a board deck and the ERP.
Then tie each number to a line the CFO already reports: days to close for a finance agent, cost per resolved request for a support agent, time to quote for a sales agent. In our experience, the teams that pick their number before the build are the ones that can show a result at the end of it.
Why evals come first
An eval is a test for an AI system, and it should exist before the system does.
An eval set is a list of real cases from the company’s own work, each with its right answer, plus a pass mark agreed in advance. The eval set should be written before anything gets built, by the people who know the work. For one firm, the client gathered real questions from its own teams, locked them in when the contract was signed and agreed the pass mark with us, including cases where the right answer was to refuse someone without access.
Three habits keep evals useful, and the first is a regression gate: re-run the evals on every change, such as a new model, a new prompt or a new data source, and block any change that lowers the score. Second, score a sample of live work on a schedule, because real cases change over time. Third, add every failure found in production to the set, so it gets harder to pass.
Everything else in this paper depends on evals, because they work like the internal controls an auditor already checks. They tell you when an agent is ready to go live, and they let you change models safely: when a cheaper or better one arrives, the evals show quickly whether anything got worse.
Exhibit 3
How an eval framework works
Before the build
On every change
In production
What it costs to run
A demo costs almost nothing, but production has running costs, and they should be budgeted per task.
The cost of a demo is a few hours of someone’s time. Production has running costs: fees for using the model, the people who check exceptions, upkeep of the software and its evals, and licenses for the tools around it. Budget each agent per completed task, set a ceiling before it goes live, and compare it with what the same task costs today.
Evals are also how that cost comes down: once a cheaper model passes the same eval set, repetitive work can move to it while the most capable models stay on the hard cases. The cost per task falls without anyone re-checking everything by hand.
Then look at the software contracts around the work, because once a company’s data sits in one place, licenses bought tool by tool usually overlap. Software vendors are changing how they charge as agents take over work that used to need a paid user license. At each renewal, put three things in writing: your own agents’ right to use the product, a cap on usage charges, and pricing that doesn’t rise when agents replace users.
A roadmap for the first year
Plan the first year as one sequence with gates, starting in finance, instead of a string of pilots.
We’d start in finance, because its work repeats and every number has to match something else, which makes it easy to write evals for. The first quarter goes on foundations: a baseline for each workflow, agreed definitions and the first eval sets. The first agents go live in the second quarter, on reconciliations and accruals.
The gates keep the plan honest: an agent moves into production only when it passes its eval set and costs less than its budget per task. The next workflow starts only once the last one is stable, and by the end of the year the board sees cost per completed task next to the savings, workflow by workflow.
Exhibit 4
A first year, with gates
- First quarterFoundationsA baseline for each workflow, agreed definitions and the first eval sets
- Second quarterFirst agents go liveReconciliations and accruals
- Rest of the yearThe next workflowsEach one starts only once the last one is stable
- End of the yearThe board sees the resultCost per completed task next to the savings, workflow by workflow
Gate: an agent moves into production only when it passes its eval set and costs less than its budget per task
What we expect, and what to ask
Most back-office automation at leading mid-market companies runs as saved software, with agents called only when a test fails
End of 2028
Auditors ask to see eval results alongside other IT controls
End of 2027
Cost per completed task appears in mid-market board packs next to headcount
End of 2027
A monthly close of five working days or fewer is normal for mid-market companies
End of 2027
Five questions to ask before anything goes live
- Is the work saved as software, and does it read the same definitions as everything else?
- What’s the baseline, and which numbers will show it worked?
- What’s in the eval set, who wrote the right answers, and what’s the pass mark?
- What does each completed task cost, and what’s the ceiling?
- Where does it sit on the roadmap, and what has to be true before the next workflow starts?
Glossary
- Agent
- Software that carries out multi-step work on its own, with a person approving the steps that matter
- Eval
- A test for an AI system: real cases with known right answers, a way to score each output, and a pass mark
- Eval set
- The collection of cases an AI system is tested against, written before the build and added to over time
- Regression gate
- A rule that re-runs the eval set on every change and blocks any change that lowers the score
- Baseline
- How long a piece of work takes, what it costs and how often it goes wrong before anything changes
- Cost per completed task
- Everything it takes to finish one piece of work with AI, including model usage and human review
- Saved run
- The software and test kept after an agent completes a task once under supervision
- Semantic layer
- One agreed definition for each business metric, with a named owner, read by every report and agent
How to cite
Imtisal Qadir and Talha Shaikh, “After the demo,” Outpost, October 2026, https://www.outpost-us.com/research/after-the-demo.
