Why one AI answer is not a measurement

AI answers change from one run to the next

The same prompt, sent to the same AI engine, does not reliably return the same brands. One screenshot of one answer tells you what happened once, not where your brand stands.

In January 2026, SparkToro and Gumshoe.ai published a crowdsourced test: 600 volunteers ran 12 prompts through three AI tools a combined 2,961 times. Their conclusion: there is “a <1 in 100 chance that ChatGPT or Google’s AI, if asked 100X, will give you the same list of brands in any two responses”, and roughly one in 1,000 runs before two lists come back in the same order ^[1].

The vendors say the same thing in their own documentation. OpenAI writes that Chat Completions “are non-deterministic by default” ^[2]. Anthropic writes that “even with temperature set to 0, the results will not be fully deterministic” ^[3].

Why the answers vary

Two layers of variation stack up: the model itself, and the live sources an AI search engine retrieves for each request.

• The model. Researchers who ran five models ten times on the same tasks, with settings meant to be deterministic, saw accuracy vary by up to 15% across runs ^[4]. Thinking Machines Lab sampled one prompt 1,000 times at temperature 0 on an open model and got 80 distinct completions ^[5].
• The sources. Ahrefs tracked more than 43,000 keywords for a month and found that an AI Overview has a 70% chance of changing from one observation to the next, with 45.5% of cited sources entirely new between consecutive observations ^[6]. Profound compared the domains cited for the same prompts one month apart and found 40% to 60% of them had changed, depending on the engine ^[7].
• The question. Real buyers do not type your test prompt. In the SparkToro study, 142 human-written prompts for the same need had an average semantic similarity of 0.081 ^[1].

What still means something

Presence across many runs is a usable signal. Position in a single list is not.

The same SparkToro data shows a hospital that appeared in 69 of 71 answers (97%) but came first in only 25 of them ^[1]. The authors conclude that measuring this “percent visibility” is “(probably) a reasonable way” to know how prominent a brand is in an AI’s consideration set. Rank position, on the other hand, moves almost every time.

That leads to a simple rule: report how often you are named across a fixed set of questions and runs, never “we ranked #2 in ChatGPT” based on one try.

How to measure AI visibility so the numbers hold up

Fix the protocol, repeat each question, report margins of error, and set a noise floor before comparing anything.

• Fix the question set. Choose your buying questions once, validate them, and replay exactly the same list at every measurement.
• Repeat every question. At ZivRank, each question is asked 5 times on each of 5 engines: 50 questions give 1,250 observations per measurement.
• Separate outcomes. Being named in the answer and appearing only as a cited source are different results. Count them separately, and keep questions that already contain your brand name out of the totals.
• Publish a margin of error. A rate without a confidence interval says nothing about whether a change is real.
• Set a noise floor first. Run the full protocol twice with nothing changed. Our noise floor is ±15%: a smaller variation is not reported as a result.
• Keep the raw answers. Every number should trace back to timestamped answers someone else can check.

The full protocol and its limits are published on our methodology page.

What to ask a vendor that tracks your AI visibility

• How many times is each question asked, on each engine, at each measurement?
• Is the question set identical from one measurement to the next?
• Are “named” and “cited as a source” counted separately?
• What is the noise floor, and how was it measured?
• Can I get the raw answers behind each number?

If the answer to most of these is no, the dashboard shows activity, not a measurement.