How to Measure AI Visibility Without Fooling Yourself

I have written elsewhere about why the tools in this category disagree with each other. This is the constructive half: if the vendor numbers are unreliable, what does a defensible measurement programme actually look like?

The answer borrows nothing from SEO reporting and everything from experimental design. That is an unwelcome answer because it is more work and produces smaller claims. It is also the only one that survives contact with a statistician.

Start by abandoning the question you want answered

The question everyone asks is “what is our AI visibility?” It has no answer, and it is worth being precise about why rather than treating this as pessimism.

Visibility would be your appearance rate across the population of relevant queries, weighted by how often each is asked. Two of those three terms are unobservable. Providers do not publish query logs. No panel captures private chat sessions at scale. You cannot enumerate the population and you cannot weight the sample.

What you can answer:

• Given this specific set of prompts, at this cadence, on this model, what proportion of responses mention us — and with what uncertainty?
• Did that proportion change after we made a specific change, by more than sampling noise would explain?
• Did it change on prompts we targeted but not on prompts we did not?

Narrower, defensible, and sufficient to make decisions. The third question is the one that separates measurement from storytelling, and almost nobody asks it.

Designing the prompt set

Everything downstream depends on this and it is where most programmes go wrong by trying to be comprehensive.

1. Small and fixed. Twenty to forty prompts you wrote and can justify individually. A prompt set that grows over time makes every historical comparison invalid, and vendors add prompts constantly.
2. Stratified by intent, not by keyword. Group by what the person is trying to do — evaluate options, solve a problem, check a fact, compare two named things. Different intents behave differently at the retrieval gate and averaging across them hides everything.
3. Include prompts you expect to lose. A set containing only queries you might win is not a measurement instrument, it is a highlight reel. Deliberate inclusion of hard cases is what lets you detect genuine improvement.
4. Hold out a control group. Reserve a third of prompts you will never optimise for. Movement in the treated group that also appears in the control is not yours — it is the platform changing.
5. Version the set. If you must change it, start a new series. Never edit prompts in a running experiment.

How many runs you actually need

This is arithmetic, not judgement, and it is the part that makes people uncomfortable because the numbers are large.

Mention is a Bernoulli trial. To DETECT A CHANGE you need
enough samples in both periods.

Approximate n per group for two-proportion comparison,
80% power, 95% confidence:

  detect 30% → 40%   (10 pts)   ≈  360 runs per group
  detect 30% → 35%   (5 pts)    ≈ 1,400 runs per group
  detect 30% → 32%   (2 pts)    ≈ 8,700 runs per group

Read that again. To reliably detect a five-point move
you need roughly 1,400 observations before and after.

With 30 prompts that is ~47 runs per prompt per period.

Anything less and you are not measuring a change.
You are observing variance and narrating it.

Two honest consequences. First, weekly reporting on small prompt sets is statistically meaningless — you do not accumulate enough observations in a week. Monthly or quarterly is the realistic cadence for claims about change. Second, you should design for large effects. If your intervention cannot plausibly move appearance rate by ten points on the prompts you targeted, you will not be able to prove it worked, and you should decide whether it is worth doing on other grounds.

What to record for every single run

Aggregates are where evidence goes to die. Store the raw observation.

timestamp            UTC, to the second
prompt_id            stable reference into your versioned set
prompt_text          the exact string sent
surface              api | consumer app | which one
model_id             exact identifier, including version if exposed
temperature          and any other sampling parameters
response_text        the complete raw response, stored
mentioned            boolean, by your stated rule
mention_position     ordinal within the response
sentiment            your classification, plus who or what classified it
cited_urls[]         every URL, in order
search_performed     did the system retrieve at all, if observable
run_index            which repetition of this prompt this is

Without response_text you cannot re-analyse when your
definition of "mention" changes. And it will change.

That last note is from experience of the general pattern: every measurement programme revises its definitions in the first quarter. If you only stored booleans, your entire history becomes incomparable the moment you improve your rules.

Define “mention” before you count anything

Write it down, in advance, with edge cases resolved. Otherwise the definition drifts toward whatever produces a nicer number.

Case	Counts as?	Why the decision matters
Named in prose, no link	Brand mention, not citation	Different value. Track separately
Linked in footnote, unnamed in text	Citation, not brand mention	User may never see the name
Named as what to avoid	Mention, negative	Counting this as a win inverts the truth
Position nine of eleven	Mention, low prominence	Record ordinal or you cannot see quality change
Product named, brand not	Decide in advance	Common and genuinely ambiguous
Competitor comparison where you lose	Mention, negative context	Sentiment classifier must handle this

And if you use a model to classify sentiment, measure the classifier. Hand-label two hundred responses, compare, and report its accuracy alongside your results. An unvalidated classifier is an unknown error term applied to every number you publish.

The annotation log

A time series without annotations is decoration. Maintain a dated log of everything that could plausibly affect the numbers, whether or not you caused it.

• Model version announcements from every provider you query
• Your own content publications and material edits, with URLs
• Technical changes: robots.txt, structured data, site architecture, hosting
• Competitor launches and notable coverage you become aware of
• Anything you notice behaving differently, even without explanation

When a discontinuity appears — and it will — the log is the only thing standing between you and a fabricated causal story. Discontinuities in this data are usually the platform. Assuming otherwise is how teams take credit for a deployment and then cannot repeat it.

First-party signals, which nobody sells you

What to instrument:

• Crawler hits by user agent, by URL, over time. Which pages are being fetched by which AI systems, and whether that is rising or falling. A page that stops being crawled will stop being cited.
• Referral traffic from assistant domains. Separate this from organic search in your analytics. It is the only direct evidence of citation converting to a visit.
• Landing page distribution of that referral traffic. Which of your pages assistants actually send people to is frequently not the pages you expected, and it is the strongest signal about what is working.
• Whether your robots.txt permits the crawlers you want. Worth verifying rather than assuming, and worth deciding deliberately rather than by default.

A programme that fits in a working week per quarter

1. Write 30 prompts. Stratified by intent, a third held as control, versioned, documented.
2. Define mention and sentiment in writing, with the edge cases from the table above resolved.
3. Run each prompt 40+ times per quarter, spread across days and times rather than in one burst.
4. Store every raw response. Not aggregates.
5. Report proportions with confidence intervals, treated and control side by side.
6. Maintain the annotation log continuously, not retrospectively.
7. Read your server logs monthly for crawler and referral patterns.
8. Claim nothing the control group does not support.

What this costs you politically

I should be honest that this approach is harder to sell internally than a dashboard.

You will report ranges where colleagues want points. You will say “no detectable change” where a vendor tool would have shown a reassuring upward line. You will occasionally have to explain that a large apparent gain was a model update and belongs to nobody. None of that is what people want from a quarterly review.

What you get in exchange is that when you do report a real effect, it is real — and you can repeat it. In a field this noisy, being the person whose numbers hold up under questioning is worth considerably more than being the person with the nicest chart. It just takes a few quarters for that to become obvious to everyone else.

Why the commercial tools cannot do this for you is covered in why AI visibility tools disagree, and what you are attempting to measure is set out in how retrieval actually selects your page.

Frequently asked questions