Refactoring Orgs

One of our largest customers for Agentic Voice just switched to a lower tier. When asked why, the person in charge told us they wanted their human staff to take more calls. Here's a customer who loves our fully autonomous AI agent product. They signed up early. They got really excited. We built everything they needed. Yet, they just decided to not use our product as much any more.

These things happen, especially because Agentic Voice is still early on the adoption curve. But after thinking about this for a while, I think there may be a bit more to it. Here it comes:

From a customer's perspective, switching from a callcenter to AI agents handling everything autonomously isn't an easy transition. When someone calls your hotline and an AI answers, it can get messy. It's hard to verify that this works better than a human call-center, for every call. Verifiability seems to be a problem.

Let me rephrase that into a more general theory: AI capabilities are likely to accelerate first in domains where progress can be verified cheaply - code, math, closed-loop optimization. In domains where verification is expensive - taste, judgment, messy human systems, calls - it will lag. The two curves will diverge sharply for a while.

That divergence may be something you want to keep in mind when making strategic decisions about new AI products:

• If the product sits in a verifiable-output industry, the productivity numbers are about to move fast, along with the layoffs and your margins. A small team of generalists can move mountains here. You'll see a lot of new customers, and churn will be low. Your product will become sticky quickly, and it will look like it almost sells itself.
• If it sits in a messy-output industry, the tools will get impressive - and very complex, fast. You'll have to invest a lot of capital to make minimal progress. We're talking larger teams, more research, very expensive domain-expert engineers. Plus, you'll need sales and customer success to convince people to use it. It's going to become very capital-intensive. But the numbers may not move. Or at least, it may take a long time for them to move. You may still see a lot of interest in the market, and some new customers, but also very high churn rates (50-70% YOY), and relatively low revenue.

Products in a messy-output industry may still get C-level support. Someone will justify the investment, either because they don't see the alternative, don't know anything but the messy-output industry, or argue you'll win precisely because you're going to invest more than your competition.

I like challenges, but when it comes to strategic AI product decisions, it's better to avoid new AI products in a messy-output industry. It may also help to keep your eyes open for new AI product ideas in a verifiable-output industry. At the very least, you should be very clear about what type of industry you're in before you make the decision.

Verifiability may also be the reason why many AI organizational change initiatives stall. There are verifiable-output jobs and tasks, but there are also messy-output jobs and tasks. The mistake is assuming every job is uniformly one or the other. Most jobs contain both. For example, engineering and sales operations clearly sit on different sides of the curve. The reason AI adoption is spiky within firms may not be the culture alone. I think more often than not it's verifiability. Ask which parts of your process a machine can grade, and you have your answer.

Verifiability is going to be a problem for a while.