A Model That Knows the Answer Says It the Same Way Five Times
Here is a fact worth sitting with. A model that actually knows the answer says the same thing every time you ask, worded differently. A model that is confabulating gives you a different name, date, or number each time, with identical confidence in its voice. That instability is not a vibe. It is measurable, and it costs nothing extra beyond samples you probably already needed.
This is the method from Farquhar et al., published in Nature in 2024: sample the model several times at temperature, cluster the samples by meaning instead of by exact string, and measure entropy over the clusters. One cluster means consistent. Several clusters, roughly even, means the model is guessing.
Clustering by Meaning Is the Whole Trick
"Paris" and "It is Paris" are the same answer worded differently. Count them as two distinct answers and you smear the signal until hallucinations stop looking any different from confidence. semanticentropy clusters by content using a transparent containment check: the smaller answer's core content just needs to be mostly present in the larger one, and numbers are normalized so "300,000" and "300000" tokenize the same.
The Number
On ten labeled question groups, consistent answers average 0.08 normalized entropy. Hallucinations average 0.90. That 0.81 gap is the entire result, and it costs zero labeled examples and zero model calls beyond the five samples you already needed to draw for the check.
10 tests pass in CI across Python 3.9, 3.11, and 3.13. Swap the containment check for a real NLI model at one clean seam when you want production-grade equivalence.