Most businesses don't know if their AI is working. Here is how to actually check

One of the questions many business owners ask themselves is "Okay, but how do I know if it's actually worth it?"

And it's an excellent question. The problem is that in most businesses I enter, it's impossible to answer — not because the AI isn't working, but because no one measured what happened before it was introduced. There is no baseline for comparison. And when there is no baseline, every answer is a feeling, not a data point.

I want to break this down, because in my opinion, it's the most neglected part of the entire subject.

The first mistake: Measuring after, not before

You've already decided to introduce a new tool to the business. You let the employees use it and after a few months, you decided to "check" if it paid off?

It's hard to answer this question once there is nothing to compare it against. How long did the task take before? How many errors did it have?

Before you think about introducing a new tool or product to the business, it's important that you document what was there before; only then will you have a real way to measure whether the tool paid off for you or not.

The second mistake: Thinking "Return" is only money

Many business owners ask "How much money will this bring in". It's a legitimate question, but it misses most of the picture. Return on investment in technology can come in several different forms:

• Time returned to the team. Not in and of itself — but in what it is invested in afterwards. Two hours a day returned to a team that wastes them is not a profit. Two hours that turn into more inquiries answered, or more time with clients — that is.
• Cost savings. Fewer work hours for the same output, less hiring, less rework.
• Increased revenue. More clients handled by the same team, or a faster response that prevents a client from going to someone else.
• Risk avoided. One mistake in a contract or a quote can cost more than everything you saved that year. This is part of the return, even if it's hard to put a number on it.

I ask clients to decide in advance which of the four they are measuring. Not all of them — one or two. A project that tries to achieve all four at once usually achieves none of them.

The third mistake: Metrics that look good but say nothing

"The team uses it every day" is not a metric. "We did 400 queries this month" is not a metric. These are numbers that are nice to see but don't teach you if anything in the business has improved.

A real metric looks more like: "The average response time to an inquiry dropped from two days to four hours", or "The number of quotes sent with an error dropped from 1 in 10 to 1 in 40". These are things you can show someone and they immediately understand if it's good or not.

Why this is even important

When a business doesn't know how to measure, one of two things happens: either it continues to pay for something that returns nothing, or — and this is more common — it gives up on something that actually worked, because there was no way to show it.

Both of these results cost money. The second one costs more, because it also wastes the investment already made.

I also think this is what separates a real implementation from just "we bought a tool". A tool that no one measured is a tool that no one is truly responsible for. And without clear ownership, within two months the team goes back to doing things as before.

What I would do in your place

If you currently have a tool running in the business and you aren't sure if it's worth it, don't start by replacing it. Start by answering three questions:

1. Which specific task is it supposed to improve, and who is responsible for it?
2. How much time/errors/rework was in that task before it was introduced? (If there is no answer — measure it this month, even in hindsight it's worth something)
3. What is the number that if you see it in a month, you'll know it worked?

If you can't answer these three, the problem is almost certainly not with the tool.