Same Question, Monday and Tuesday
You ask for a summary of a document. It is good. The next morning you ask for the same summary, expecting to paste it into the same report, and what comes back is recognisably the same content in a different order, with a different opening line and one point missing. Nothing changed. You did not change anything.
People find this more unsettling than they expect to, and they usually reach for one of two wrong explanations: that it remembered the earlier conversation and is trying not to repeat itself, or that the model was updated overnight. Neither is happening. The real reason is simpler, and once you know it you can decide how much of it you want.
It Is Not Picking an Answer. It Is Rolling Dice.
At every step of writing a response, the model does not produce one next word. It produces a ranked list of candidates with a probability attached to each. After "the invoice is now thirty days" it might rate "overdue" very likely, "old" plausible, "past" possible, and a long tail of others as vanishingly unlikely.
Then something picks one. If it always picked the single highest-ranked candidate, you would get identical output every time. Most systems do not do that. They sample from the distribution — usually taking the top option, sometimes the second, occasionally something further down. Do that a few hundred times in a row, once per fragment, and two runs diverge quickly even though nothing about the model changed.
Prompt “The shipment has been ___”
Medium: the default range. Natural-sounding, still on topic.
The dial is called temperature. Low temperature flattens the odds toward the favourite, so output gets predictable and, at the extreme, repetitive and a little flat. High temperature evens out the field, so more unusual choices get a look in — livelier, more varied, and more likely to wander somewhere you did not want.
When the Variation Is Fine, and When It Is Not
The same behaviour is a feature in one column and a defect in the other.
Variation helps
- Drafting, where you want options rather than one answer
- Brainstorming and naming, where sameness is the failure
- Rewriting a sentence three ways to pick the best
- Anything creative, where the second-choice word is often better
- Getting unstuck — a different phrasing can restart your thinking
Variation hurts
- Extracting a figure, date, or name from a document
- Classifying or routing — the same input must sort the same way
- Anything downstream software has to parse
- Repeated documents that should look identical every time
- Comparing outputs to judge whether a change improved things
The second column is where this stops being a curiosity. If you are running the same classification across two thousand records, a system that sorts an identical input differently on Tuesday than it did on Monday is not usable — and you may not notice for weeks, because each individual answer looks reasonable.
How to Get More Consistency
In order of how much they help, in our experience:
Constrain the output shape
"Reply with one of exactly these three labels" removes almost all room for variation, because there is barely any distribution left to sample from. Wherever a task can be turned into a choice from a fixed list, do it — this does more than any setting.
Turn the temperature down
If you are using the API, this is a parameter. Near zero for extraction and classification. Most chat interfaces do not expose it, which is one of the honest reasons to move a repeated task out of a chat window.
Give an example of the exact output you want
Two or three worked examples in the prompt narrow the range more effectively than any amount of describing. The model matches the pattern rather than inventing a format each time.
Ask for the answer only
"Return the date, nothing else" prevents the preamble and the closing paragraph, which is where a lot of the visible variation lives even when the substance is stable.
The Version of This That Actually Costs Money
There is a subtler failure worth naming, because it catches people who have otherwise done everything right. You test a prompt, it works beautifully five times, you put it into production, and a few weeks later somebody notices the output has been wrong for a while.
Nothing broke. Your five tests happened to land on five good samples. The sixth kind of input was always going to produce something worse, and you had no way to see that from five runs. This is why testing a prompt on twenty varied real examples — deliberately including the awkward ones — tells you something that testing on five clean ones cannot.
If a repeated task matters, build in the checks that catch drift:
- Test on twenty real examples, chosen to include the messy cases
- Run the same input twice and compare — if the answers differ meaningfully, tighten the prompt before shipping
- Spot-check a sample regularly once it is running, not just at launch
- Have somebody own the output quality, because unowned quality drifts
The Short Version
Key takeaways
- The model produces a ranked list of possible next words and samples from it — the variation is a dice roll, not a memory or a change.
- A little randomness is what stops generated text reading as repetitive and flat, so it is not a defect to be eliminated.
- It is genuinely fine for drafting and genuinely a problem for extraction, classification, and anything software parses.
- Constrain the output shape first, then lower the temperature, then give worked examples.
- Test a repeated prompt on twenty varied real examples. Five clean ones will pass and tell you nothing.
Why does ChatGPT give different answers to the same question?
Because it samples from a probability distribution rather than always taking the single most likely next word. At each step it has a ranked list of candidates and picks from it with some randomness. Repeat that a few hundred times per response and two runs diverge, even with identical input and an unchanged model.
What is temperature in AI?
A setting that controls how much randomness goes into that choice. Low temperature concentrates the odds on the most likely option, producing predictable and sometimes repetitive text. High temperature flattens the distribution so less likely candidates get chosen more often, producing more varied and more unpredictable text.
How do I make AI give me the same answer every time?
You can get very close but rarely to a guarantee. Constrain the output to a fixed set of choices, set temperature near zero if you have API access, include two or three worked examples, and ask for the answer only with no surrounding commentary. Constraining the output shape does more than everything else combined.
Does AI remember my previous questions?
Not between separate conversations. Within one conversation it appears to remember because the whole history is re-sent with every message. Start a new conversation and it knows nothing — which is also why telling it to "answer the same way as last time" has nothing to attach to.
Is inconsistent output a sign of a bad AI model?
No — it is inherent to how these systems generate text, and better models are not more deterministic by default. What separates a well-built system is not the absence of variation but whether the task was set up to constrain it where consistency actually matters.