All posts

Does ChatGPT give the same answer to everyone? Only 28% of brands repeat

welaunch·November 18, 2026

No. Does ChatGPT give the same answer to everyone? Running the same prompt four times, only 9 of 32 brand appearances (28%) survived every run, and batch processing on OpenAI's servers means even temperature-zero, identical prompts can split apart after the first hundred words. One answer is a sample, not a measurement.

Those brand numbers come from welaunch's test of 434 live AI answers to buyer questions in seven software categories, run 27 Sep 2026 in the US. The technical half of the story comes from published inference research. Both point the same way.

The explanation you've already heard (and why it's only half true)

Ask why ChatGPT varies and someone will say "temperature." The model samples from a probability distribution over the next word, temperature controls how adventurous that sampling is, so turn it down and you get the same text back. Clean story.

It does not survive contact with a test. Researchers fed the same prompt to a model a thousand times with temperature pinned at zero and all randomness disabled, and got around eighty different answers instead of one. The sampling dial was off. The variation stayed.

So temperature is one source of variation, not the source. If you have been explaining answer drift to your team as "it's random," you have an incomplete model of what is happening, and it will mislead you the moment you try to measure anything.

Turn the randomness off and it still changes

The shape of the divergence in that thousand-run test is the interesting part. The completions were word-for-word identical for the first hundred or so words, then split and wandered in different directions.

That pattern tells you the cause is not a dice roll at the start. It is something small that accumulates. The model generates one word at a time, each word conditioned on everything written so far. For a hundred words the probabilities line up the same way. Then one borderline choice goes the other way, and from that point the context is different, so every subsequent choice is made in a different world.

For anyone checking whether their product gets mentioned, this matters: if a brand name appears later in the answer, it falls in the part of the response most exposed to drift.

The real reason: your answer depends on the strangers next to you

The dominant cause is batch sensitivity. Thinking Machines Lab's September 2025 write-up identifies it as the dominant cause of nondeterminism in LLM inference.

Here is the mechanism. To serve millions of people efficiently, ChatGPT bundles incoming requests together and processes them as a group, a batch. Your question rides along with whatever other questions arrived at the same moment. The GPU does the arithmetic for the whole batch at once.

Floating-point addition is not perfectly associative. Adding the same set of numbers in a different order can give a slightly different result in the last decimal places. And the batch size changes the exact order that billions of additions happen in, which changes the rounding errors, which can change which word the model selects next.

So the answer you get depends in part on how many other people hit the servers in the same instant as you, and what they asked. Not their content, just their volume. That is a genuinely strange property for a product that looks like a search engine.

The other things quietly changing your answer

Batching is the invisible cause. There are visible ones too, and they are easier to control.

ChatGPT's conversation history and memory are read by the model, so two identical questions asked in different chats are not really identical. The text you typed matches. The context wrapped around it does not. If you have ever asked ChatGPT about your own product in your own account and been pleased with the result, memory is a likely explanation, and your buyers will not have it.

Then there is the account itself. The free tier, a paid tier, and a custom GPT can each start from different hidden instructions and sometimes different underlying models. Two people asking the same question through different doors are not running the same test.

Practical version: check in a fresh chat, logged out or with memory off, and understand that you are still only sampling one tier of one product at one moment.

Does everyone get the same answer from ChatGPT?

Now the measured side. In welaunch's test of 434 live AI answers (27 Sep 2026, US), each buyer question was asked four times on each engine and every brand name in every answer was recorded.

Of 32 brand appearances in ChatGPT's answers, 9 repeated in all four runs. That is 28%. Nearly three quarters of the brands ChatGPT named for a question did not show up every time the same question was asked.

A second measure says the same thing from a different angle. Comparing the set of brands ChatGPT named in one run against the set from another, across the seven prompts that named any brand, the mean Jaccard similarity was 0.54. Jaccard is the overlap between two sets divided by their union: 1.0 means the same brands every time, 0 means no shared names. Roughly half the recommendation list changes between runs of the same question.

So no, everyone does not get the same answer. Two buyers typing the same words minutes apart can see different shortlists, and neither of them is seeing a mistake.

Does ChatGPT give the same answers to everyone, or just mostly the same?

"Mostly" is doing a lot of work in that question. The honest answer is that the stable core is smaller than people assume, and it is not unique to ChatGPT.

Perplexity, tested the same way in welaunch's 27 Sep 2026 run, named 6 of 21 brand appearances in all four runs, 29%, with a mean Jaccard similarity of 0.57 across the five prompts that named any brand. Close enough to ChatGPT that you should treat the variance as a property of this class of product, not a quirk of one vendor.

What this implies in practice: there is usually a small set of brands that always appear, the ones the engine is confident about, and a larger rotating cast that appears sometimes. If you are in the rotating cast, a single screenshot can tell you that you are visible or that you are invisible, and both readings are correct about that one run and wrong about your actual position.

EngineBrand named in all 4 runsMean Jaccard similarityReddit citation rateAvg sources per answer
ChatGPT9 of 32 (28%)0.546% (7 of 110)5.8
Perplexity6 of 21 (29%)0.5731% (32 of 103)10.1
GeminiNot statedNot stated20% (22 of 110)9.2
Google AI ModeNot statedNot stated40% (44 of 111)3.4

Source: welaunch's test of 434 live AI answers to buyer questions in seven software categories, 27 Sep 2026, US.

Why do AI visibility tools give different results for the same prompt?

Because they are sampling the same unstable thing you are, just with more automation.

Reading this because you need customers?

Paste your URL. See what ChatGPT, Perplexity and Google AI answer when your buyers ask, who they name instead of you, and which pages they read. Then we do the work to get you onto those pages.

Free scan, no signup, no card.

An AI visibility tool sends a prompt to ChatGPT, reads the answer, and records which brands appear. That request lands in a batch alongside strangers' requests, exactly like yours. If the tool asks once a week and your brand sits in the rotating cast, the tool will report you present some weeks and absent others, with no change in the world to explain it.

Two tools running the same prompt at different seconds on different accounts will disagree for the same reason, and both will have logged a real answer. Neither is lying. They sampled different draws from a distribution where, in welaunch's test, only 28% of ChatGPT brand appearances held across four runs and only 29% of Perplexity's did.

The question to ask a vendor is therefore not "is your data accurate" but "how many runs is each number built on, and can I read the raw answers." A percentage from one run per week is a percentage with an enormous error bar that nobody drew.

Is this a bug?

It is a side effect of an engineering decision, and it is fixable. Horace He and the team at Thinking Machines Lab showed that batch sensitivity can be engineered away by rewriting the math so it gives the same result regardless of batch size.

That tells you two things. First, the cause is understood, not mysterious. Second, nobody has to live with it forever. Whether the large providers ship batch-invariant kernels by default, and what that costs in throughput, is their call, and I have no numbers on the performance trade-off to give you.

Until then, plan around the variance rather than hoping it disappears. Your measurement approach should survive a fix as well as the status quo.

Does this happen with every type of question, or just some?

Engine behaviour is not uniform, and the citation data in welaunch's 27 Sep 2026 test makes that plain. Across 110 ChatGPT answers to buyer questions, it cited 5.8 sources on average and gave no source at all in 7 of them. Perplexity cited 10.1 sources across 103 answers and gave no source in none of them. Gemini cited 9.2 across 110 answers with 11 sourceless. Google AI Mode cited 3.4 across 111 answers with 21 sourceless.

Where those sources come from diverges even more sharply. Reddit appeared in 6% of ChatGPT answers (7 of 110), 20% of Gemini's (22 of 110), 31% of Perplexity's (32 of 103) and 40% of Google AI Mode's (44 of 111).

An answer built on ten retrieved sources has different stability properties from an answer built on three, or from one built on none at all, where the model is drawing on training weights alone. Questions that trigger heavy retrieval anchor to whatever the index returned that minute. Questions that do not are pure generation, with all the drift that implies. So expect different question types in your category to behave differently, and measure each of them rather than generalising from one.

Can you force ChatGPT to be consistent?

Partly. You can remove the controllable sources of variation, not the structural one.

What you can control: start a fresh chat so conversation history and memory are not feeding extra context into an apparently identical question. Hold the tier and the interface steady, since the free tier, a paid tier and a custom GPT can start from different hidden instructions and sometimes different models. Use the API with fixed parameters if you want fewer moving parts.

What you cannot control: the batch. Batch size changes the order of billions of additions and therefore the rounding, and you have no say over who else is sending requests at that moment. Pinning temperature to zero does not fix it, as the thousand-run test showed.

So the realistic goal is not a repeatable answer. It is enough samples that the distribution becomes visible.

What this actually means for you

Stop treating a single ChatGPT answer as a result. It is one draw. At a 28% four-run repeat rate for ChatGPT brand appearances in welaunch's 434-answer test, a screenshot of your product being recommended and a screenshot of it being ignored can both be true in the same hour.

Three habits that follow from the data:

Run each buyer question several times, on different days, and record every answer rather than a summary. Trend lines across runs are the only honest unit of measurement here.

Track the sources, not only the mentions. Citation rates differ hugely by engine, from 3.4 sources per answer on Google AI Mode to 10.1 on Perplexity, and the pages an engine keeps returning to are more stable than the brand list it builds from them.

Treat any one-off audit, yours or a vendor's, as an anecdote. Ask how many runs are behind every number you are shown.

This is the shape of the problem welaunch was built around: it asks your buyers' questions on ChatGPT, Perplexity, Gemini and Google AI several times a week, keeps every answer in full so you can read what each engine actually said, finds the pages those engines cite, and drafts the reply, listing or page that gets you onto one of them. The drafts go out from your own accounts, after you review and press submit.

FAQ

Is every ChatGPT answer unique?

Not every one, but far more vary than people expect. With temperature at zero and randomness disabled, a thousand runs of one prompt produced around eighty different answers, and those answers were identical for about the first hundred words before diverging.

Is everyone's ChatGPT different?

Yes, in a few ways at once. Memory and conversation history feed context into your questions that other people's accounts do not have, and the free tier, paid tiers and custom GPTs can start from different hidden instructions and sometimes different underlying models. On top of that, batching makes the output depend partly on who else is querying at the same moment.

Does the 28% repeat rate measure the same thing as technical nondeterminism?

No, it measures the visible consequence. The 28% figure, from welaunch's 27 Sep 2026 test, counts how many brand appearances in ChatGPT answers survived all four runs of the same prompt. Batch sensitivity, memory and retrieval differences all feed into that number without being separable from it.

Does busier server traffic mean more answer variation?

That is the implication of the mechanism, since batch size changes the order of operations and therefore the rounding that can flip a word choice. But I have no measurement comparing quiet hours to peak hours, so treat it as a plausible consequence rather than a tested result. The safer approach is to spread your checks across different times of day.

Can you force ChatGPT to be consistent for testing?

You can remove some variation: fresh chats, memory off, one fixed tier, fixed API parameters. You cannot remove batch sensitivity, because you do not control the other requests processed alongside yours. The practical fix is repeated sampling rather than a search for one true answer.

Why do two AI visibility tools report different numbers for my brand?

Because each one sent its prompt at a different moment and got a different draw. With roughly half the brand list changing between runs on ChatGPT (mean Jaccard similarity 0.54 in welaunch's test), two honest tools can produce two different reports from the same prompt. Compare how many runs each number is based on before deciding which to trust.

Sources

Related on welaunch

Do buyers find you, or your competitor?

Paste your URL. See what ChatGPT, Perplexity and Google AI answer when your buyers ask, who they name instead of you, and which pages they read. Then we do the work to get you onto those pages.

Free scan, no signup, no card.

Part of Measuring it honestly, in the guides on getting your SaaS named in AI answers.

Does ChatGPT give the same answer to everyone? Only 28% of…