Distinguishing Real Change from Sampling Noise in LLM Search Outputs

Author's Note

Large language models with web search produce variable outputs: the same prompt, submitted repeatedly to the same system on the same day, yields different responses each time. This variability poses a practical problem for anyone monitoring them — when a brand's visibility moves from 40% to 35% week over week, it is difficult to know whether something real has changed (a model update, new content, a reindex) or whether the difference reflects sampling noise alone.

That question had never been rigorously answered in public. We conducted this research to present a concrete way to separate a genuine shift from sampling noise, and a caution that at the run-counts most tools use today, a lot of what looks like a 'change' is just chance.

Tom Zu, Founding Engineer

Large language models with web search produce variable outputs: the same prompt, submitted repeatedly to the same system on the same day, yields different responses each time. This variability poses a practical problem for anyone monitoring them — when a brand's visibility moves from 40% to 35% week over week, it is difficult to know whether something real has changed (a model update, new content, a reindex) or whether the difference reflects sampling noise alone.

That question had never been rigorously answered in public. We conducted this research to present a concrete way to separate a genuine shift from sampling noise, and a caution that at the run-counts most tools use today, a lot of what looks like a 'change' is just chance.

Tom Zu, Founding Engineer

Large language models with web search produce variable outputs: the same prompt, submitted repeatedly to the same system on the same day, yields different responses each time. This variability poses a practical problem for anyone monitoring them — when a brand's visibility moves from 40% to 35% week over week, it is difficult to know whether something real has changed (a model update, new content, a reindex) or whether the difference reflects sampling noise alone.

That question had never been rigorously answered in public. We conducted this research to present a concrete way to separate a genuine shift from sampling noise, and a caution that at the run-counts most tools use today, a lot of what looks like a 'change' is just chance.

Tom Zu, Founding Engineer

Glossary

Runs - times submitted the same prompt to an LLM model

Visibility - percentage of runs where the model recommended certain brand

Citation Share - percentage of one citation’s appearance count over that of all citations

Key Takeaways

Key Exhibits

Smallest detectable change in brand visibility, by runs per batch. Below the line, a difference between two batches is indistinguishable from re-running the same batch twice.

• sampling noise shrinks with the square root of the number of runs, so precision gets expensive fast.

• The sampling noise converges toward ~1 percent, you can never rule out noise.

Experiment Setup

We submitted the same six buyer-style questions to ChatGPT and Gemini web search over and over across a single day. Every response was collected the way a real user would see it: through the web interface, logged out, with no conversation history.

• 7200 trials in a single day — 600 runs of each prompt on each engine, on the same day to rule out temporal effects

• Six product-research prompts, three industries — B2B SaaS, consumer retail, and financial services; 

• One broad and one long-tail question per industry (e.g., "What are the best travel credit cards?" vs. "…for someone who travels internationally 2–3 times a year and wants lounge access?")

Outline

Section I: Introduction — An open question in the literature

Prior work documented that LLMs are non-deterministic and that their behavior drifts across versions, but not how many runs it takes to measure any of it. Published evaluations use anywhere from tens to thousands of trials, usually without justification. This paper closes that gap for the two metrics that matter commercially: brand visibility and citation share.

Section II: Experiment Design — The statistical framework

We treat each prompt submission as an independent trial. Our metrics are brand visibility (a Bernoulli proportion) and citation share (a ratio of citation counts). It defines a detection threshold τₙ: the 95th percentile of the difference between two independent samples of size n drawn from the same distribution, i.e. a critical value under the null hypothesis at two-sided α = 0.05. Thresholds are estimated with a 10,000-iteration nonparametric two-sample bootstrap at every n from 5 to 600. The i.i.d. A binomial sampling model is validated empirically, which licenses a closed-form power analysis separating the false-positive boundary from the larger effect size needed for 80% power.

Section III: Results — Brand visibility

The smallest trustworthy visibility change shrinks with run count, and it depends on the brand's baseline. The power analysis shows the stricter bar, a real 5-point shift at 100 runs is caught only 11% of the time. The good news is that ordering is cheap even when magnitude isn't, brand rankings reach 0.90 correlation between independent samples by 80 runs in every engine and prompt type.

Section III: Results — Citation Share

The same analysis for the sources AI engines cite. Top domains become measurable after ~50 runs, but for 83% of long-tail domains, the noise envelope is wider than their entire citation share even at 600 runs. To track changes in citation share, even a very small change ±1 pt requires 100 runs.

Sections IV–V: Conclusion

Practical guidance for anyone comparing two batches of LLM outputs: configure your sample size carefully with the noise thresholds in mind, treat sub-threshold differences as noise, and expect long-tail citation claims to be unverifiable. 

Abstract

Large language models with integrated web search produce stochastic outputs: the same prompt submitted to the same surface on the same day yields different responses across independent trials. Any empirical study that compares two batches of trials, whether across model versions, prompts, days, surfaces, or other conditions, must therefore distinguish a real underlying difference from sampling noise. Sample sizes in published LLM evaluation studies vary widely, however, and the relationship between sample size and minimum detectable effect for downstream commercial-visibility metrics has not been characterized empirically with public methodology. We address this question for two metrics of practical interest, brand visibility and URL citation share, by submitting six product-research prompts (spanning three industries and two specificity levels) to two web-grounded LLM access surfaces (ChatGPT 5.5 via chat.openai.com and Gemini 3.3 Flash via gemini.google.com, both logged-out free accounts), with 600 trials per (prompt, engine) cell, on a single day. We define a detection threshold τn as the 95th percentile of |∆|between two independent samples of size n-drawn from the same trial distribution, and we estimate τn across n∈{5,...,600} via a 10,000-iteration nonparametric two-sample bootstrap. The empirical bootstrap standard error for brand visibility agrees with the closed-form binomial standard error within a mean absolute relative error of 1.69% across 22 non-saturated brand-cells, consistent with an independent- and-identically-distributed binomial sampling model. Within-day independence holds: mean lag-1 autocorrelation of -0.011 and median time-of-day mean-visibility spread of 0.33 pp. Visibility detection thresholds depend strongly on baseline visibility. At n= 100, τn peaks at 14.00 pp for brands nearˆ p= 50%, in close agreement with the closed-form prediction of 13.86 pp, and falls to a median of 4.00 pp for brands within 10 percentage points of saturation. Citation-share thresholds partition sharply by tier: for long-tail URLs (share < 0.1%), the threshold τn exceeds the URL’s observed share itself at n = 600 for 83% of the 60 long-tail URLs observed, meaning the sampling-noise envelope is wider than the share being measured and a same-magnitude shift could not be confidently distinguished from chance within the experimental regime studied here (n≤600). Brand-ranking stability, measured via Spearman ρ between two samples of size n, reaches median ρ≥0.90 at n≤80 in every (engine, prompt- type) cell. We use these results to inform sample-size selection in empirical LLM-output studies.

Acknowledgements

With thanks to Zachary Bergman, whose work on the experimental pipeline and analysis made this research possible.