LLM Hallucinogenics

Everyone knows by now; AI models (LLMs) hallucinate--present things as facts while being utterly wrong. We laugh at the more recent "is there a sea-horse emoji?". But in fields like healthcare, hallucinations can be outright dangerous.

LLMs have gotten far though, the newer and larger models hallucinate far less. Plus, I think people are getting better at sensing when to ask AI to 'look something up', or to avoid certain tasks entirely. But still, there's a huge need for prevention--or at the very least detection--of LLMs hallucinating. And it turns out, LLMs are less of a 'black box' as they used to be.

Why they hallucinate

There are two ways to explain why a person turns out the way they do. There's the therapist's answer; 'here's what happened to them, the experiences that shaped them'. And there's the neurologist's answer; 'here's what's physically going on in the brain that produces the behavior'. Both help build an understanding of what's truly going on.

Explaining AI hallucinations was previously exclusively done using the therapist method. So let's start there. An LLM's "childhood"--the data it was raised on--gets blamed in a few common ways:

• Bad data diet: it couldn't grow up factual because some of what it was fed simply wasn't true (Reddit is a big part of the average LLM's diet 🤷‍♂️).
• Malnourishment: it doesn't have the information it needs. It might not know the latest World Cup result yet, but it can still feel confident answering, based on the pattern of every other tournament it has seen.
• Imbalanced diet: information in the world is lopsided. The model reads a thousand descriptions of how the well-documented case works and only a handful of the obscure one, then takes the pattern from the common case and stamps it onto the rare one where it doesn't fit.

As mentioned, AI as a 'black-box' is changing as we're able to look under-the-hood more. Or to stretch the healthcare analogy: we now have the AI equivalent of MRI machines and neurologists—meaning we can give the other kind of answer.

Mapping out the whole inner workings of an LLM's mind will be its own article, but the part that matters here is that these models grow certain skills on their own. A vision model, for instance, taught itself something called edge detection--it roughs out a kind of pencil sketch of an object's outline to help work out what it's looking at, without anyone telling it to. And there seems to be a self-taught ability that could be directly effecting hallucinations.

The model's sense of familiarity

Anthropic's interpretability team found that refusal is actually Claude's default behavior--a circuit that's "on" unless something switches it off. Only when a "known entity" feature fires does that default refusal get suppressed. In their example, Claude answers questions about basketball player Michael Jordan without hesitation, but refuses to answer about a fictional "Michael Batkin"--until researchers artificially activated the known-entity feature, at which point Claude confidently, and wrongly, claimed Batkin plays chess. (Anthropic, 2025)

This gives us the first 'neurological' cause of hallucination; the familiarity signal misfires. A topic or entity registers as familiar to the LLM but doesn't actually have the specifics, and that vague sense of "oh yeah, I know this" is enough to flip the refusal off--even though the knowledge behind it was never there. And that empirically makes sense to me. Listing out a specific iPhone's feature specs is likely to hallucinate without direct context because it's familiar with all the iPhones, but might mix up the exact details.

Which raises the obvious question; why is that familiarity signal so willing to fire when it shouldn't? Part of it is that the signal is just imperfect. But part of it is, again, how we raised it. We trained these models on a mountain of tests--multiple choice especially. And as you'll remember from school, "I don't know" is typically not one of the options. And no answer scores the same as a wrong answer; zero. So the smart move, for a student or a model, is to always put at least something down--25% > 0%. We rewarded confident guessing over honest abstention, over and over, and the models learned the lesson exactly.

So, bringing the two answers together;

• The neurologist's version: a familiarity heuristic that misfires and hands out false confidence in the gaps.
• The therapist's version: a training regime that punished saying "I don't know."

LLM's factuality under new light

But I'm skimming over a critical insight here; LLMs gauge factuality based on familiarity. The junction between "I don't know" and "Here's the answer..." is solely how familiar it is with it. Which I'd say is a little different than how us humans treat confidence in answering questions. I could be familiar with iPhones and get asked about a specific's model's features--but I'm not going to answer because I know that's easy to get mixed up. And that's not even mentioning that familiarity doesn't entail factuality.

Familiarity vs Factuality

To me, this is the least recognized challenge in AI in healthcare. Being familiar with something doesn't mean you know the facts. Us humans suffer from the same; the fluency heuristic.

The fluency heuristic is a mental shortcut where the brain equates the ease of processing or recalling information with its truth, value, or validity.

But humans can have the luxury of being curated a niche and vetted information diet; doctors go to accredited institutions while the largest data hungry LLMs consume the internet at large. Would you trust a doctor that got half of their education from Reddit?

Provenance

I'm being crude and perhaps extreme with the analogies, I believe LLMs (and Reddit) can be exceptional and have significant utility. And truthfully, I wouldn't mind my doctor being part of online community discussions--I'd celebrate it. But when the scalpel is in their hands, I'd want to be sure that their reasoning is based on peer reviewed papers--not an opinion of an anonymous Redditor. Provenance matters for humans too, but it's a harder problem for LLMs. Solving that for LLMs is non-trivial issue--more on that in another article.

Conclusion

AI/LLMs are amazing, we can now converse with all the information the internet has--but it can make mistakes. The things it learned can be wrong, it can confuse one topic with a similar one, or it just simply can't know but feels confident enough anyway. With recent research, we're able to see how a LLM is judging confidence by familiarity--not factuality. Which is a difficult problem, especially considering the healthcare industry that's rapidly adopting AI.

In the next article I'll be exploring the tools we currently have to mitigate hallucinations and how you could detect when a model is unfamiliar but proceeds anyway. I'll be exploring provenance in AI in more depth in the meantime as well, so if you have any ideas to share in that space, feel free to reach out!