Most AI visibility tools report one number as if ChatGPT were one product.
It isn’t.
In Petra Labs testing, the same brand swung by 32 percentage points across three OpenAI access surfaces on the same day using the same prompt.
Most AI visibility tools measure only one slice of ChatGPT, not the full range of channels your customers actually search on. If you are optimizing for AI search, you need to account for every surface where those searches happen.
In our 900-trial study, visibility differed materially across paid ChatGPT, free ChatGPT, and the API. A tool reporting a single “visibility” number is usually measuring one of those three at most.
What "accuracy" actually means for an AI visibility tool
When SEO professionals talk about accuracy in Google rank tracking, the question is largely settled: there's one Google, rankings can be measured, and tools converge on the same number.
AI visibility is different. The model produces stochastic outputs. Results vary by access surface, by region, by user history, and by the time of day. Five axes determine whether a tool's reported number actually reflects what your buyers see:
Sample size. Does the tool run enough trials per prompt to distinguish signal from random variation?
Prompt selection. Are the prompts representative of how buyers actually phrase their queries, or hand-picked to flatter the brand?
Surface coverage. Does the tool sample logged-in ChatGPT, logged-out ChatGPT, and the API, or just one?
Run-to-run variance. How much does the same prompt vary across runs on the same surface? Is the variance documented?
Source attribution. Does the tool track which sources LLMs consume during research, or only the final citations they credit?
A tool can be technically accurate on one of these axes and badly wrong on another. The result is a reported number that looks confident but doesn't predict what customers experience.
This blog focuses on the gap with the biggest measurable impact: surface coverage. The other four are subjects for separate analysis.
The accuracy gap most tools share: single-surface sampling
We tested three access surfaces of OpenAI's GPT-5.2, each representing a distinct mode of interaction with the same underlying model:
Logged-in paid ChatGPT
Logged-out free ChatGPT
The Responses API
Surface | Think of it as |
|---|---|
Logged-in paid ChatGPT | Consumer power user |
Logged-out free ChatGPT | Anonymous public web user |
The Responses API | Infrastructure layer powering apps and agents |
These surfaces may differ in token usage, system prompts, search backends, geographic signals, and temperature settings. We ran 300 identical trials per surface on a single day (February 26, 2026), using one multidimensional product research prompt. Web search was forced on every surface; no system prompts, no conversational context, no personalization.
The three surfaces diverged on every dimension we measured.
Brand visibility swings by access surface
Brand | Logged-In | Logged-Out | API | Max gap |
|---|---|---|---|---|
Blizzard | 30.0% | 30.0% | 62.3% | 32.3% |
Line | 48.3% | 0% | 18.3% | 48.3% |
Vōlkl | 0% | 47.0% | 17.7% | 47.0% |
Renoun | 18.0% | 15.3% | 0% | 18.0% |
Brand-visibility correlation across pairings
Surface pairing | Pearson r |
|---|---|
Logged-In ↔ Logged-Out | 0.891 |
Logged-Out ↔ API | 0.815 |
Logged-In ↔ API | 0.711 |
Top-5 brand overlap (Jaccard similarity)
Surface pairing | Top-5 Jaccard |
|---|---|
Logged-In ↔ API | 0.25 |
Logged-In ↔ Logged-Out | 0.43 |
Logged-Out ↔ API | 0.67 |
The top-5 most-recommended brands on logged-in ChatGPT and the API overlap by 25%. Three of the top five brands on paid chat are not in the top five via the API.
A tool that samples one ChatGPT surface and reports "your visibility is 30%" is reporting visibility on one slice of a model that behaves like three different products.
Is this just random noise?
We confirmed it isn't. Within-surface split-half analysis: we divided each surface's 300 trials into two halves of 150 and computed per-brand visibility independently in each half. Within-surface RMSE across the halves ranged from 4.5% to 7.1%. Cross-surface RMSE between the full 300-trial datasets ranged from 10.8% to 16.1%. The surface effect is 1.5 to 3.6× larger than run-to-run randomness. And that's the conservative comparison, since per-half estimates are noisier than full-sample estimates.
Put simply: ChatGPT behaves more differently across surfaces than it does across repeated runs on the same surface.
In other words: the variance between logged-in ChatGPT and the API is meaningfully bigger than the variance within either surface across hundreds of trials of the same prompt.
What this means practically
If your buyers are mostly on paid ChatGPT, optimization gains there may not show up in API tests.
If your tool tracks the API only, it may report your brand as invisible when it is being recommended to thousands of chat users daily.
Worst case: niche brands. One brand in our test (Renoun) appeared in 15–18% of chat trials and zero API trials. A tool sampling only the API would report that brand at 0% visibility, indistinguishable from a brand with no AI presence at all.
Full methodology and dataset: How Access Surface Shapes LLM Search and Citation Behavior.
How Petra approaches accuracy
Petra Labs is a full-service AEO partner, not a self-service tracker. But because measurement underwrites every intervention we run, we built our internal approach around the same accuracy principles this guide describes:
Multi-surface sampling: Every prompt we track is sampled across logged-in ChatGPT, logged-out ChatGPT, and the API, with the surface disclosed on every report.
Multi-LLM sampling: Petra Labs can track prompts across multiple LLMs: ChatGPT, Claude, Google AI Overviews, etc.
Sample size sized to the question: Prompts that drive client decisions are tracked with enough trials to distinguish 5–10 point movements from noise.
Discovery-driven prompt mapping: We build a 2,500–3,000 prompt map per client from real buyer language, not from marketing team intuition.
Interim and final citation tracking: We report both, so clients can tell content-quality problems from discoverability problems.
Daily monitoring: No weekly snapshots. Citation tracking refreshes every 24 hours.
Published methodology: Our underlying research, including the Access Surface paper this guide draws from, is open and replicable.
We do not think every brand needs Petra. Some buyers fit a self-service tool, some need a monitoring layer, some need full-service. The relevant question is not which tier. It is whether the tier you choose is built on accurate measurement or convenient measurement.
A separate comparison of where each tier sits: Petra Labs vs. Self-Service AEO Tracking Tools.
FAQ
Why does my AI visibility tool show different numbers than what I see when I ask ChatGPT myself?
The model's outputs vary stochastically. Same prompt, same surface, same minute can produce different brand recommendations. Your tool may also be sampling a different access surface than the one you are testing on. If your tool tests the API and you check on logged-in ChatGPT, you can expect double-digit visibility differences for the same brand.
How many prompts should I be tracking?
That depends on the breadth of your category. covers branded comparisons, unbranded category prompts, regional variants, and intent layers from research to comparison to purchase.
Is API-side visibility important if my customers use ChatGPT, not the API?
Increasingly, yes. Agent products (developer tools, custom GPTs, embedded ChatGPT-style features inside other software) consume the API on behalf of users. As that ecosystem grows, your "buyer" may be querying via the API without ever visiting chat.openai.com.
Why don't my AEO investments seem to show up in my tool's numbers?
Three possibilities: (a) the intervention worked on a surface your tool does not sample, (b) the change is real but smaller than your tool's noise floor, or (c) the intervention is still being indexed and has not propagated. Tools that track interim citations alongside final ones distinguish (a) and (c) clearly.
Can I just run my own ChatGPT prompts to check visibility instead of buying a tool?
For spot checks, yes. For decisions, no. A single manual query is one trial. Drawing inference from one trial in a stochastic system is like predicting tomorrow's weather from yesterday's temperature. Tools are valuable because they run hundreds of trials per prompt per day. Bad tools are valuable for the same reason but produce misleading averages.
Does Google AI Overviews behave like ChatGPT?
Different model, different ranking logic, different document index. AI Overviews and ChatGPT can both pull from your domain, but they are separate optimization targets. A tool that conflates them into one "AI visibility" score is hiding more than it shows.