Why AI pilots die before production
Most AI pilots die not because the technology failed but because a pilot is optimised for the wrong thing. A pilot is designed to demonstrate that something is possible; production requires something to be reliable, integrated, exception-handling and adopted. Those are different projects, and success at the first predicts almost nothing about the second.
Written by Josh Soderblom, founder of ikonic303.
Enterprises have access to genuinely capable models. Most still cannot get them into daily operation. Here is where it actually breaks.
1. The pilot ran on the clean 20%
Pilots are built on tidy examples — the standard order, the well-formed document, the customer who filled the form in properly.
Production is the other 80%. The scanned PDF at an angle. The order that says “same as last time.” The customer whose name is spelled three ways in your database. The exceptions are not edge cases; they are the workflow. A system that handles the clean cases and dumps the rest on a human has moved the bottleneck and called it a win.
The tell: nobody can tell you the exception rate. If the answer to “what percentage does it not handle” is a shrug, the pilot has not met production.
2. Nobody owned the integration
The pilot ran standalone. Production means it lives inside a CRM someone configured in 2019, an ERP with a customisation nobody documented, and a scheduler with an API that returns HTTP 200 on failure.
That work is unglamorous, hard to estimate, and routinely descoped because it does not demo. It is also where most of the actual engineering is.
3. The process was automated instead of redesigned
The workflow you automate is nearly always the workflow as it accumulated, not as anyone would design it. Three of its steps exist because of a system replaced in 2018.
Automating a bad process makes the bad process faster. Almost every deployment turns up at least one step that should stop existing rather than be automated — and finding it requires permission to change how the work is done, which a pilot never has.
4. Nobody adopted it
The quietest killer. The system is live. It works. People go round it.
Usually for a mundane reason: it is slower for the person doing the job, even though it is faster for the company. It needs a login they do not have. It fails on one job type so they stopped trusting it. They were never asked and they resent it.
A system that is technically live and quietly unused reports as a success and delivers nothing.
5. The vendor left at go-live
The six weeks after launch are where deployments are decided — the seasonal case, the file format that changed, the person who joined and was never trained.
If the engagement ended at go-live, all of that is now your problem, and the vendor's success metric was met at the exact moment your risk began.
What they have in common
Every one is a failure of ownership across the boundary. Someone was responsible for making the thing work, and someone else was responsible for it working in real life, and the gap between them is where projects die.
That gap is the entire argument for embedding. Not because embedded people are cleverer, but because there is nobody to hand the hard part to.
How to avoid it, whoever you use
- Do not run a pilot. Pick the smallest workflow that is genuinely valuable and take it all the way to production. A completed small thing teaches you more than a successful pilot of a big thing.
- Make the exception rate a headline number, tracked from day one.
- Get permission to change the process before anyone writes code.
- Put one real operator in the build, from the start, not at UAT.
- Contract the six weeks after launch into the engagement. Whoever built it should be there when it meets reality.
- Define success as usage, not as delivery. “Live” is not a metric. “Doing 90% of jobs through the new system in week eight” is.
What the ten weeks look like → · How an engagement runs →
What ten weeks looks like · AI agents that do work · How an engagement runs
Get in touch
Email solutions@ikonic303.dev to start with the measurement. ikonic303 is based in Colorado and works with clients nationwide.