Measuring AI
Your AI usage numbers look good. They are not telling you if AI is delivering results.
Picture the quarterly review. Someone pulls up a dashboard showing AI adoption percentages: Licenses activated
Prompts run
Average sessions per user
Everyone nods. The numbers look healthy.
Nobody asks what those numbers actually mean for how work is getting done. And nobody in the room asks the question that actually matters:
Has anything about how this organization operates changed since people got access to these tools?
Most organizations default to what is easiest to count
Tool logins
Prompt volume
Time saved on specific tasks
These are the metrics that show up first because they are the metrics the tools themselves generate.
Prosci's study, Evaluating AI Success: Metrics that Differentiate Successful AI Implementations (March 2025), confirms what most practitioners already sense: time efficiency is the most common measurement category organizations reach for when evaluating AI success.
These metrics answer one question: are people touching the tools?
They do not answer a different and more important question: is work transforming?
An employee who runs the same three prompts every week and reports time savings is using AI.
An employee who has fundamentally rethought how she approaches a deliverable because AI changed what is possible is adopting AI.
Those are not the same thing, and a usage dashboard cannot tell you which one you are looking at.
The gap between usage and genuine adoption is where most AI investments quietly stall.
There is something else measurement approaches are not seeing.
If you are only counting logins to the sanctioned tool, you are missing the fact that people may be using AI through unsanctioned channels.
Your adoption dashboard shows low usage, but the reality may be that people are adopting AI actively, just not the way you planned and not with the tools you are tracking.
Shadow AI is not defiance.
It is a signal that your sanctioned tools are not matching real work needs. And your measurement approach is not seeing it.
What the Research Says About What Works
The organizations that succeed with AI implementation do not measure adoption in one dimension. They measure across multiple dimensions simultaneously.
Prosci's research on AI implementation found that organizations measuring success across multiple dimensions, not just speed or efficiency, succeed at significantly higher rates than those relying on a single measurement category.
The principle is intuitive once you see it: time savings tells you about efficiency.
It tells you nothing about:
the quality of work
the level of trust people place in the tools
how teams are collaborating
the kind of capability the organization is building over time.
Measuring in one dimension gives you one signal. Measuring across multiple dimensions gives you a picture of if adoption is real or cosmetic.
This is not a new insight.
Change practitioners have long understood that a single outcome metric misses much of what determines long-term success. But AI tools generate usage data so easily and automatically that it becomes the default measure, and organizations forget to ask if usage is actually what they should be tracking.
The practical stakes are real.
Leaders who are trying to understand where AI creates new capacity need multi-dimensional data to see the full picture.
And leaders who are evaluating if roles need to change require that same data to avoid decisions based on surface-level usage numbers that tell them very little about what is actually happening in the work.
One-dimensional measurement does not just miss the story. It gives leaders false confidence that they understand something they do not.
The Role-Level Blind Spot
There is another dimension many measurement approaches miss entirely: different organizational levels care about fundamentally different things when it comes to AI adoption.
Executives focus on strategic outcomes:
moving the business forward
creating competitive advantage
driving the numbers they report to the board.
Individual contributors care about something entirely different. They want AI to make their work:
more creative
more interesting
more manageable.
Better work, not just faster work.
Underneath that, many are weighing a question they may not say out loud: does AI make me more valuable here, or does it make me replaceable?
These are also the people closest to the daily reality of how AI fits into the work, and the most likely to tell you if the tools are genuinely useful or just one more thing they have been told to use without a clear reason.
Managers sit in the middle, focused on coordination, workflow, and figuring out if AI is helping or disrupting how their teams operate.
A measurement approach built around what executives care about will miss the signals that tell you whether adoption is real at the level where work actually happens.
If your dashboard reports strategic outcomes and efficiency gains, but you have no way to see if individual contributors are actually integrating AI into how they think about their work, you are measuring leadership’s priorities, not the change itself.
The signals that matter most live closest to the work.
And the people closest to the work are often the least likely to show up in the metrics an executive reviews.
Signals, Not Metrics
The organizations that get this right are not chasing metrics. They are watching for signals. Signals that tell them:
if the change is taking hold
where it is stalling
where to intervene
A metric implies a target to hit: 80% adoption by Quarter 3.
A signal tells you a story about what is actually happening:
people in a specific job are finding new applications for AI beyond what the training covered
while people in another job have stopped experimenting after Week 2.
Metrics give you a number for a dashboard. Signals give you something to act on.
The distinction matters because it changes what you pay attention to and when to intervene.
Metrics invite patience: we are at 60%, we need to get to 80%, give it another quarter.
Signals invite action: something is working here, something is not working there, and we can see it now while there is still time to adjust.
If your AI adoption dashboard is full of usage data and empty of work transformation signals, you are measuring the tool, not the change.
The tool is the easy part to track.
How people are actually changing their work is harder to see and harder to measure than tool usage. But it is the only thing that tells you if your AI investment is producing lasting value or just producing activity.
My methodology was designed around this measurement problem, not around the usage dashboards most organizations reach for.
Let’s talk about what AI adoption really takes. Contact me today.