Why AI pilots die before production

When an AI pilot dies, the cause is often not the technology. A pilot is built to show that something is possible. Daily use requires it to be reliable, connected to your other systems, able to handle the odd cases, and actually used. Those are different projects, and success at the first says little about the second.

Written by Josh Soderblom, founder of ikonic303.

A pilot is a small trial: a demo on a handful of examples. A pilot can work and still never become part of the day. Here are five places it breaks, whether you are building it yourself or paying someone to.

1. The pilot ran on the clean examples

Pilots are built on tidy examples — the standard order, the well-formed document, the customer who filled the form in properly.

Real work is everything else. The scanned PDF at an angle. The order that says “same as last time.” The customer whose name is spelled three ways in your database. The exceptions are part of the workflow, not a footnote to it. A system that handles the clean cases and dumps the rest on a human has moved the bottleneck and called it a win.

The tell: nobody can tell you the exception rate. If the answer to “what percentage does it not handle” is a shrug, the pilot has not met real work yet.

2. Nobody owned the integration

The pilot ran on its own. Daily use means it has to work with a CRM someone set up years ago, accounting software with a custom tweak nobody wrote down, and a scheduling tool that reports success even when something failed.

That work is unglamorous, hard to estimate, and easy to cut because it does not show well in a demo. It is also a large share of the real work.

3. The process was automated instead of redesigned

The workflow you automate is nearly always the workflow as it accumulated, not as anyone would design it. Some of its steps may exist only because of a system you replaced years ago.

Automating a bad process makes the bad process faster. Look for the step that should stop existing rather than be automated. Removing it requires permission to change how the work is done, which a pilot usually does not have.

4. Nobody adopted it

The quietest killer. The system is live. It works. People go round it.

Usually for a mundane reason: it is slower for the person doing the job, even though it is faster for the company. It needs a login they do not have. It fails on one job type so they stopped trusting it. They were never asked and they resent it.

A system that is technically live and quietly unused reports as a success and delivers nothing.

5. The vendor left at go-live

The weeks after launch are where it is decided — the seasonal case, the file format that changed, the person who joined and was never trained.

If the work ended at go-live, all of that is now your problem, and the vendor was finished at the exact moment your risk began.

What they have in common

Every one is a failure of ownership across the boundary. Someone was responsible for making the thing work, and someone else was responsible for it working in real life, and the gap between them is where projects die.

That gap closes when one person owns both sides. It is the argument for a builder who stays until the thing is in use. It is also why an owner who builds a small tool for their own work starts with an advantage: there is nobody to hand the hard part to.

How to avoid it, whoever you use

  1. Do not run a pilot. Pick the smallest workflow that is genuinely valuable and take it all the way to daily use. A completed small thing teaches you more than a successful pilot of a big thing.
  2. Make the exception rate a headline number, tracked from day one.
  3. Get permission to change the process before anyone writes code.
  4. Put one person who does the job in the build, from the start, not at the final test.
  5. If someone is building it for you, put the weeks after launch in the agreement. Whoever built it should be there when it meets real work.
  6. Define success as usage, not as delivery. “Live” is not a measure. Something like “90% of jobs go through the new system by week eight” is — with your own number and date.

A small task of your own is the right place to start. In our AI classes for business owners, owners work in Claude Code on tools for their own businesses. If you would rather have it built, see small AI tools for one task.

What the ten weeks look like → · AI classes for business owners →

What ten weeks looks like · AI agents that do work · AI classes for business owners · AI tools built for one task


Get in touch

Email solutions@ikonic303.dev to get the next AI class date, to talk about marketing, or about one AI tool or job you want built. ikonic303 is based in Colorado, and classes are held in the Denver metro.

ikonic303 home · AI classes · what we built · free Anthropic courses · contact