The AI Product Gap

I've spent about three years now with the best AI tools money can buy open in front of me every working day. Not as a tourist — as a daily driver, the way you'd use a code editor or a terminal. I build agentic infrastructure for a living, so these tools aren't a curiosity to me; they're the workbench. And when you live in something that long, you stop seeing the marketing and start seeing the seams.

The most consistent thing I've found isn't a model limitation. Models are improving faster than almost anything I've watched in my career. The consistent thing is a gap — the distance between the demo that sold me and the product that survived contact with my actual work. I've started calling it the AI product gap, and I think learning to see it is one of the most useful skills an AI product person can have right now.

I want to be careful here. This isn't a teardown of anyone's product, and I'm not naming names, because the gap isn't about one company being lazy. It's structural. It shows up everywhere, including in things I've built. It's a discipline for buyers and builders, not a list of villains.

"Looks Done" Is Not "Is Done"

The demo shows you the happy path. That's its whole job. Somebody chose the input, chose the example, chose the moment to hit record. What you see is the product at its best, on the one road that was paved before the camera turned on.

Real work doesn't travel the paved road. Real work is the edge case, the malformed input, the integration that turns out to be stubbed. I've lost count of the times I've adopted something that looked finished, leaned on it, and then discovered — usually by reading the source or hitting the one input that mattered to me — that the impressive part was a façade over a TODO. The feature wasn't lying, exactly. It was demoed, not lived in. Nobody had spent day 30 inside it, so nobody had found the place where it quietly does nothing.

This is the difference I now anchor on: "looks done" and "is done" are completely different claims, and a demo can only ever prove the first one. The second one is proved by use. By the boring Tuesday where you needed it to work and it either did or it didn't.

I learned this the expensive way long before AI, shipping enterprise software where a launch that breaks is a strategy that failed. You don't get credit for a beautiful build that falls over in production. The launch is the proof. AI products are subject to the exact same physics; we've just gotten very good at making the build look like the launch.

Reliability Is a Feature, Not a Footnote

Here's the one that's hardest to see from the outside: the status page and the experience are two different products.

A status page measures uptime in the aggregate. It tells you the service was technically reachable 99-point-something percent of the time. What it doesn't tell you is whether it was reachable for you, mid-task, at the moment you'd built a thought on top of it. The downtime you actually feel isn't the kind that trips a monitor. It's the request that hangs while you're holding a fragile chain of reasoning in your head, the timeout that lands right as you were about to ship, the degraded response that's technically a 200 and practically useless.

I've felt that gap enough times that it changed how I build. When reliability is someone else's footnote, it becomes your problem at the worst possible moment. So I went local-first with my own stack — I run my own agent runtime and my own memory platform on hardware in my house, zero cloud. People assume that's an ideology thing. It isn't. It's that I'd rather own my reliability than rent someone's demo. When my overnight chain has to be right by morning with no one watching, "the vendor had a rough night" is not an answer I can use. Reliability isn't a footnote on an AI product. It's a feature — arguably the feature — and the products that treat it as one are the ones that hold up on day 30.

Features That Don't Compose

The subtlest version of the gap is the product where every feature works alone and none of them work together.

You can feel it when you try to chain things. Feature A is great. Feature B is great. But A's output isn't shaped like B's input, the state doesn't carry across, the thing you did in one place is invisible in the next. Each piece was clearly built and demoed in isolation — each got its own happy path, its own recording — but nobody lived in the seam between them, so nobody felt how badly they refuse to hold hands.

Composition is where lived-in products separate from demoed ones. When you actually use a system every day, the seams are most of your experience; you spend more time moving between features than inside any one of them. A product designed by people who use it tends to have features that compose, because the friction in the seams is the friction they feel too. A product designed to demo well tends to have a gallery of impressive, lonely capabilities. The whole is less than the sum of its parts, and you only discover that after you've committed.

The Lens I Actually Use

So how do you judge an AI product if you can't trust the demo? You move the question.

The demo answers what does this do in minute 3? The only question I care about anymore is what does this do on day 30 of real use? Day 30 is past the honeymoon. Day 30 is after you've hit the edge case, felt the downtime that doesn't show on the status page, and tried to make the features compose. Day 30 is where "looks done" either became "is done" or quietly didn't. Almost everything that matters about an AI product is invisible at minute 3 and obvious by day 30 — which is exactly why the demo is recorded on day zero.

As a product person, that's the lens I bring and the bar I hold my own work to. I don't trust a feature because it looks done in a recording; I trust it because it survived a month of me actually needing it. That bias — toward the thing that holds up over the thing that demos well — is the whole discipline.

And I want to end on the honest balance, because cynicism is cheap and wrong here. The frontier is genuinely, thrillingly fast. The gap is shrinking; things that were façades a year ago are real now. The point of seeing the gap isn't to sneer at the demo — it's to make better calls. To buy the product that survives contact with your work, and to build the product that survives contact with someone else's. Ship the thing that holds up, not the thing that demos well. That's not pessimism. That's the job.

If you're building AI products and you want someone who judges them by day 30, not minute 3, I'm at reed@grainlabs.io.