AI agents that do work
A useful business agent takes a defined input, performs a defined job, produces a defined output into a system people already use, and hands off visibly when it is not confident. Most business AI fails all four — it is a chat window bolted to a knowledge base, which is a search box that sometimes makes things up.
Written by Josh Soderblom, founder of ikonic303.
The four properties
1. A bounded job
Good: “Read the inbound RFQ email, extract the line items, match them to catalogue SKUs, flag anything it cannot match with confidence.”
Bad: “Help the sales team.”
The bounded version can be tested, measured and improved. The unbounded version can only be demoed. If you cannot write the job as one sentence with a verb and an object, it is not ready to build.
2. Real input, real output, existing systems
If someone has to open a new tab to get the value, the value is optional. Optional things get skipped in week three when everyone is busy.
The agent should be triggered by something that already happens — an email arriving, a job closing, a form submitted — and should put its output where the work already lives.
3. A confidence boundary
The single most important property, and the one most often missing.
The agent must know what it does not know, stop, and hand to a human — visibly, with the reason attached.
A system that handles 95% correctly and silently guesses the other 5% is worse than the person it replaced, because those errors are now invisible until they reach a customer. The person it replaced would have asked.
Design rule: it is always better to escalate than to guess. The cost of an unnecessary escalation is thirty seconds of somebody's attention. The cost of a silent wrong answer is a customer.
4. An audit trail
In month four something will be wrong and you will need to know what the agent did and why. If it cannot show its inputs, its decision and its output for a specific record on a specific day, you cannot debug it and you cannot defend it.
Where agents genuinely earn their keep
- Unstructured to structured — turning email, PDFs and phone notes into records. The strongest use case there is.
- Triage — deciding what a human needs to see, and giving them the context alongside it.
- Extraction at volume from documents arriving in a dozen formats.
- First drafts of repetitive documents, for a human to approve.
- Watching — noticing the thing nobody has time to check, like quotes with no follow-up in ten days.
Where they do not belong
- Irreversible or expensive decisions with invisible errors. Final pricing, compliance, commitments. A design rule, not a technology limit.
- Anywhere a rule already works. If an
ifstatement solves it, use theifstatement — it is cheaper, faster, testable, and it does not drift. - As a substitute for fixing a process. Automating a bad workflow makes it faster and harder to see.
- As a general assistant nobody was asked about. Adoption is not a rollout email.
The failure mode to design against
Silent degradation. Deterministic code breaks loudly — errors, alerts, someone gets paged. Model-driven steps do not break. They get slightly worse. Nobody notices for weeks, and by then a hundred records are subtly wrong.
So build in:
- A measure of its own output quality, sampled continuously.
- A threshold at which it stops and asks rather than continuing to be quietly wrong.
- A human sampling routine — someone checks twenty outputs a week. Boring, cheap, and it is the control that actually catches drift.
How to start
Pick one job. Bounded, high-volume, currently done by a person who resents it, where being wrong is recoverable and visible. Build that. Measure the exception rate and the error rate honestly.
Then decide whether to do a second one — with real data from your own business rather than from a vendor's slide.
AI agents and automation → · Why pilots die →
AI agents and automation · Why AI pilots die · Measuring the return on an automation
Get in touch
Email solutions@ikonic303.dev to start with the measurement. ikonic303 is based in Colorado and works with clients nationwide.