Why AI Visibility Tools Disagree With Each Other

There are now dozens of platforms selling AI visibility measurement, and I have yet to read a comparison of them that examines how any of them actually measure. The roundups rank features, pricing and interface. One of the better ones — a fifteen-tool review — includes the line that there is still no reliable way to estimate true prompt volume in AI search, and then proceeds to rank fifteen tools that report numbers derived from prompt volume.

That is the whole problem in one sentence, published by the industry itself and then ignored.

I am not arguing these tools are worthless. I am arguing that the number on the dashboard is a statistic with properties nobody discloses, and that treating it as a measurement rather than as an estimate with unquantified error is producing real budget decisions built on sand.

What these tools actually do

Stripped of branding, every platform in this category runs the same loop.

1. Compile a set of prompts believed to represent your market
2. Send each prompt to one or more assistant APIs, on a schedule
3. Parse each response for brand mentions and cited URLs
4. Aggregate into metrics: mention rate, share of voice, citation count
5. Chart over time

Every step contains a choice the vendor made and does not publish.

Step one is the entire ballgame. Everything downstream is arithmetic on a sample whose construction determines the answer.

Six reasons the numbers diverge

1. The prompt set is invented, not observed

Traditional keyword tools sample real search behaviour — imperfectly, with modelling, but the underlying data is queries people actually typed. No equivalent exists for assistant conversations. The providers do not release query logs, and there is no clickstream panel that captures private chat sessions at meaningful scale.

So vendors generate prompts: from your keyword list, from a model asked to imagine what buyers would ask, from competitor analysis, from customer interviews. All defensible. None observed. Your “share of voice” is your share of a hypothetical conversation somebody constructed.

Change the prompt set and the number changes. That is not a bug in the tool — it is the definition of the metric.

2. Nobody weights the prompts

Here is a failure that would not survive first-year statistics. A prompt set of two hundred questions treats all two hundred as equally important. In reality some of those questions are asked constantly and some have never been asked by any human being.

An unweighted mean over a sample with wildly unequal true frequencies is not an estimate of the population. Appearing in 40% of prompts nobody asks and 5% of the prompts everyone asks produces a headline figure that is precisely inverted from your commercial reality.

3. Sampling is stochastic and sample sizes are small

Generation is probabilistic. The same prompt to the same model at the same temperature returns different text. That is by design, and it means a single query is one draw from a distribution.

Treat “does brand X appear” as a Bernoulli trial and the arithmetic is unforgiving.

Estimating a mention rate p with n samples.
Standard error ≈ √(p(1-p)/n)

At a true p of 0.30:

  n = 1     you observe 0 or 1. No information.
  n = 10    SE ≈ 0.145  → 95% CI roughly 2% to 58%
  n = 30    SE ≈ 0.084  → 95% CI roughly 14% to 46%
  n = 100   SE ≈ 0.046  → 95% CI roughly 21% to 39%
  n = 400   SE ≈ 0.023  → 95% CI roughly 26% to 34%

To detect a 5-point change with any confidence you need
hundreds of runs PER PROMPT. Most tools run each prompt
once or a handful of times per cycle.

Which means: a dashboard moving from 28% to 33% is,
in the overwhelming majority of cases, noise.

I would like someone to show me a vendor that displays a confidence interval. I have not seen one. Every platform in this category renders a point estimate as though it were a reading from an instrument, and clients then hold agencies accountable for movements that are indistinguishable from sampling variance.

4. You are probably not being measured on the surface your buyers use

Tools query APIs. Your customers use consumer apps. These are not the same system, and the differences are material.

	API (what tools measure)	Consumer app (what buyers use)
Model served	The one you specify	Routed by tier, load and query type
System prompt	Yours, or none	Vendor’s, long, undisclosed, changes often
Memory and personalisation	Absent	Present and per-user
Tool selection	Often forced or unavailable	Model decides whether to search at all
Conversation context	Single turn	Frequently turn three of a real conversation
Location and account signals	Datacentre	Real user, real geography

The single-turn issue deserves emphasis. Almost all monitoring measures the first question in isolation. Real usage is conversational — people refine, follow up, and reach a recommendation on turn four with three turns of context shaping it. Nobody is measuring that, because it is combinatorially hard to sample and nobody wants to admit the surface they sell reports on is unrepresentative.

5. “Mention” is defined differently by every vendor

Parsing sounds trivial and is not. Consider what should count:

• Brand named in prose but not linked
• Linked in a citation footnote but never named in the text a user reads
• Named as an example of what to avoid
• Named once in a list of eleven, in position nine
• Named as the primary recommendation in the first sentence
• Your product named without your brand, or your brand named without your URL

Some tools count all of these as one mention. Some weight by position. Some run a sentiment classifier — itself a model, with its own error rate, unreported. A negative mention counted as positive is not a rounding error; it is the opposite of the truth.

6. The system under measurement changes without notice

Model versions are updated. Routing rules change. Retrieval thresholds get tuned. System prompts are rewritten. None of it is announced, versioned or dated.

In any other measurement discipline, an uncontrolled instrument change mid-series invalidates the comparison. Here it happens continuously and the chart simply carries on, with a smooth line implying continuity that does not exist. When your visibility drops eight points overnight, the most probable explanation is that somebody at a vendor deployed something on a Tuesday.

Questions that separate a serious vendor from a dashboard

I would put every one of these to a platform before signing, in writing, and treat evasion as an answer.

1. How was my prompt set constructed, and can I see all of it? If you cannot inspect and edit the prompts, you cannot interpret the metric.
2. How many times is each prompt run per reporting period? If the answer is one, every point on the chart is a coin flip.
3. Do you report variance or confidence intervals? The answer is almost certainly no. Ask what the standard error is on a typical reading.
4. Which exact model and tier do you query, and via API or the consumer surface? Then ask whether that matches what your customers use.
5. Do you measure multi-turn conversations? Almost nobody does. Knowing this changes how much weight the number deserves.
6. How do you define a mention, and how is sentiment classified? Ask for the taxonomy and the classifier’s measured accuracy.
7. Do you log model version changes and annotate the chart? Without this, you cannot distinguish your performance from their deployment.
8. Can I export the raw responses? Not the aggregates. The actual text. If not, you are buying a conclusion you cannot audit.

How I would measure instead

None of this means give up. It means measure things that survive the noise.

• Use a small, fixed, defensible prompt set you wrote and understand. Twenty questions you can justify beats two hundred generated ones. You are trading breadth for interpretability, which is the right trade when the metric is noisy.
• Run each prompt enough times to say something. Thirty is a floor for detecting large effects. Fewer than ten is theatre.
• Report a range, never a point. If your reporting cannot express uncertainty, change the reporting rather than pretending it does not exist.
• Track citation presence separately from brand mention. They are different outcomes with different causes and different value. Conflating them hides which one you actually moved.
• Instrument your own logs. AI crawler user agents in your server logs and referral traffic from assistant surfaces are first-party, unambiguous and free. This is the most underused signal available and nobody sells it to you because there is nothing to sell.
• Annotate everything. Every model release, every content change, every deployment you know about. Without annotation a time series is decoration.

And accept the ceiling. With current data access you cannot know your true visibility. You can know whether a specific, deliberate change moved a specific, well-defined measurement. That is a smaller claim and it is the only honest one available.

What I would actually buy

A tool that lets me define my own prompts, run them at a sample size I choose, export every raw response, and annotate the timeline. In other words, an instrument rather than a verdict.

Most of this category sells the opposite: a proprietary prompt set you cannot see, a single number you cannot decompose, and a chart designed to be shown to someone who will not ask how it was made. That product exists because it is what gets bought, and I understand why it gets bought. It is still the wrong product.

If you take one thing from this: ask for the confidence interval. The reaction to that question tells you more about a vendor than any feature comparison, and it costs nothing.

The mechanism these tools are attempting to observe is set out in how retrieval actually selects your page, and the statistical design for doing this properly is in how to measure AI visibility without fooling yourself. Why the layer being measured is not the model is covered in ChatGPT is not an LLM. And there is a variable none of these dashboards state at all — the jurisdiction a query was run from, which now changes the result for legal rather than algorithmic reasons, covered in what EU regulation is doing to search and AI visibility.

Frequently asked questions