Part of the ongoing AI series at cantechit.com
Let me start with something that is going to sound like a hot take but I promise it isn’t: AI doesn’t hallucinate. Not really.
I know. Bear with me.
Ever since ChatGPT blew the doors off in late 2022, “hallucination” has been the word everyone reaches for when an LLM makes something up with absolute confidence. Scholars are now pushing back on that framing — and honestly, they’re right. Cognitive scientists and AI researchers argue the more accurate word is confabulation — a psychiatric term for when a brain (or a system that mimics one) fills in narrative gaps with plausible-sounding fiction it genuinely believes is true. It’s not lying. It’s not hallucinating in the clinical sense. It’s pattern-matching on flawed or incomplete ground truth and arriving at a confident, wrong answer.
Sound familiar? It should. Because humans do this constantly.
Every time someone doubles down on a completely false “fact” they picked up from a bad source twenty years ago, that’s confabulation. Every time a witness in a courtroom sincerely misremembers what they saw — their brain filling in details that feel true — that’s confabulation. The neurons in your brain and the mathematical structures inside a large language model are working on the same fundamental principle: reasoning forward from whatever ground truth they were trained on. If the training data is garbage, the output will be wrong. Confidently, convincingly, devastatingly wrong. The AI isn’t uniquely broken. It’s human in a very uncomfortable way.
So stop calling it hallucination. Start calling it what it is. And then — more importantly — ask yourself what that means about how much you should trust either source.
The Human Context Window: Our Last Real Edge
Here’s where I want to push back on the doomers a little, because there is one genuinely huge advantage the human brain has over every AI system that exists today: our context window is enormous.
Every LLM has a context window — the amount of information it can hold “in mind” at once during a session. GPT-4 can handle about 128,000 tokens. Claude goes up to 200,000. That sounds huge until you realize your brain is running a context window that incorporates decades of lived experience, emotional memory, embodied knowledge, real-world cause-and-effect, and deeply personal relationships — all simultaneously, all the time, all connected to each other in ways that no model can currently replicate.
Yes, an AI has access to a much larger static memory — the entire training dataset — but it has to recall and retrieve from that pool every time, with no persistent working memory across sessions by default. You carry your context window with you everywhere. It never logs out.
That’s not a small thing. That’s arguably the biggest structural advantage humans have right now, and I don’t think we talk about it enough. When I’m making a judgment call at work, I’m not just running that query against a database. I’m drawing on thirty-some years of pattern recognition, failure, intuition, and relationships. No model has that. Not yet.
The danger? We’re about to voluntarily give up the benefits of that context window by outsourcing the thinking that keeps it sharp.
We’ve Been Here Before. And We Mostly Got It Wrong.
If you want a preview of where this ends up, go back to 2008. Technology writer Nicholas Carr wrote a now-famous Atlantic Monthly cover piece called “Is Google Making Us Stupid?” — arguing that the ease of online search was rewiring how we concentrate and think deeply. It blew up. People pushed back hard. The debate raged for years.
Here’s my honest take on that debate, looking back from 2026: both sides were partially right, but they were asking the wrong question.
Google didn’t make us stupider. But it absolutely changed the nature of the critical work required. When you Googled something back in 2005, you’d get ten blue links. You had to evaluate which site was authoritative. You had to triangulate. You learned to recognize the signs of a credible source versus the signs of someone’s angry SEO-optimized blog. The old joke was “skip to page 3 — the first page is all ads.” That joke existed because people were doing the work of curation. It was like going to the library, pulling twelve books off the shelf, and deciding which one to trust. That’s not passive. That’s active, effortful, deeply human reasoning. Even if now the list of links is manipulated by an algo.
The Pew Research Center documented this debate extensively in 2010 — and one of the most interesting takeaways was that the people defending Google argued it would force people to develop better critical thinking because there was simply more information to sift. And for a while, in the right hands, that was even true.
The shift didn’t happen with Google. The shift is happening now.
Google AI Overviews: When the Library Stops Asking You to Read
Pull out your phone and search for something — anything remotely answerable — and there it is: a tidy Gemini-generated paragraph at the top of the results. Google’s AI has already done your thinking. It has synthesized the sources, drawn a conclusion, and presented it to you wrapped in a bow. No links to evaluate. No competing claims to weigh. Just: here is the answer.
A 2025 Pew study found that roughly 18% of all Google searches in March 2025 triggered an AI Overview, and that question-based searches (“who,” “what,” “why”) triggered them 60% of the time. That’s not a niche feature. That’s the default experience for the majority of how people actually phrase searches.
Here’s the thing: it is extremely convenient. I’m not going to pretend it isn’t. I use it. You use it. We all use it. The problem isn’t the convenience. The problem is that convenience is actively removing the step where you had to do critical work.
New research published in 2025 tracking the rise of AI search found that AI-answered search queries surface significantly fewer diverse sources, lower response variety, and in some topic areas, a measurable lean toward particular information ecosystems compared to traditional search. Less variety. Less exposure to competing claims. Less opportunity for you to notice when something smells off – and more opportunity for corporate and government interference in the answer.
And critically — the research showed that people believe AI answers more readily than they believe a random link. The authoritative, confident, grammatically perfect paragraph at the top of the page registers to the human brain as trustworthy in a way that a link to a mid-tier website never did. This is a real problem. The very thing that made Google-era search retain some critical thinking — the requirement to evaluate sources — is being sanded away.
The Research Is In. It Doesn’t Lie.
This isn’t just instinct. The studies are piling up and they’re not subtle.
A January 2025 study published by researcher Michael Gerlich surveyed 666 participants and found a strong negative correlation between frequent AI tool usage and critical thinking ability — mediated specifically by cognitive offloading. When you hand a cognitive task to an AI, you don’t just save time. You deprive yourself of the practice that builds the underlying skill. The muscle doesn’t fire; it doesn’t grow. Younger users (17–25) showed the highest rates of AI dependence and the most pronounced effects.
Microsoft Research and Carnegie Mellon went further in a 2025 study of 319 knowledge workers across 936 real-world AI use cases. The finding that stuck with me: higher confidence in AI was directly correlated with less critical thinking. The more you trust the tool, the less you question it. And the less you question it, the less you actually think.
Seventy-two percent of participants in knowledge tasks reported “much less” or “less” cognitive effort when using AI. Seventy-nine percent in comprehension tasks. Seventy-six percent in synthesis tasks.
Think about that for a second. Synthesis — taking disparate pieces of information and building something coherent — dropped by three quarters. That’s not productivity. That’s atrophy.
When Confabulation Has Real Consequences: The Hall of Shame
Let me give you some examples of what happens when nobody stops to question the AI. Because this isn’t theoretical. People are getting hurt, sanctioned, and embarrassed — at every level of society.
The most famous one you’ve probably heard of. In 2023, New York attorneys Steven Schwartz and Peter LoDuca submitted a legal brief in Mata v. Avianca that cited six federal court decisions. The opposing counsel couldn’t find any of them. Neither could the judge. They didn’t exist — ChatGPT had invented them wholesale: fake case names, fake docket numbers, fake quoted passages, fake procedural history. All of it. When questions arose, Schwartz did the most human possible thing: he asked ChatGPT whether the citations were real. ChatGPT confirmed they were. They weren’t. Both attorneys were sanctioned. The judge called it abandonment of their professional responsibilities. The detail that haunts me is Schwartz’s testimony: “I just was not thinking that the case could be fabricated.” That is cognitive offloading with a $5,000 fine attached.
But it doesn’t stop in courtrooms. Here are three more:
Air Canada’s chatbot told a grieving customer the wrong thing — specifically, it misrepresented the airline’s bereavement fare policy to a customer who had just lost a family member and was trying to book a last-minute flight. The customer relied on what the chatbot said, booked accordingly, and was then denied the discount. Air Canada argued in court that its chatbot was a separate legal entity and it wasn’t responsible for what it said. A Canadian tribunal disagreed. Air Canada lost. The defense of “the AI said it, not us” did not work — and it won’t in the future either.
South Africa withdrew its entire Draft National AI Policy in 2025 — just 17 days after publishing it — because the document cited fake research generated by AI. This was supposed to be a historic moment: the first African nation establishing a formal AI ethics board. Instead it became the first government in history to pull an official policy document because of AI confabulation. The communications minister was direct: the most plausible explanation was AI-generated citations included without verification. “Consequence management” was promised for those responsible.
The Chicago Sun-Times published a summer reading list in May 2025 recommending books that don’t exist — fake titles, fake descriptions, attributed to real authors who never wrote them. Out of 15 books on the list, only 5 were real. The list came from a content partner who admitted to using AI to generate it. People bought the print edition. There was no correcting that.
These aren’t edge cases. A legal scholar’s database tracking AI confabulation in judicial proceedings has catalogued over 200 cases globally — and that’s just the courtroom. The pattern across all of them is identical: someone trusted the confident, fluent, authoritative-sounding output without stopping to verify. The AI didn’t lie to them. It confabulated — exactly like a brain working from flawed ground truth. And nobody applied the critical thinking that would have caught it.
But — Some Muscles You Were Never Going to Flex Anyway
Here’s where I want to complicate my own argument a little, because I’d be a hypocrite not to.
I am never going to learn to code. I’ve made peace with that. I tried. I didn’t enjoy it. I didn’t continue. And because I didn’t continue, I couldn’t build the things I wanted to build — which was genuinely frustrating for years. AI changed that. I am now building things that are real and functional and mine. Here’s a teaser: this whole AI journey — all the reflection, all the late nights, all the agents I’ve babysitted — has quietly turned me into a builder who wants to give something back. I’ve been spending my spare time building something for the rally community. Regional events, TSD, the kind of grassroots racing that most people outside the sport never hear about — not WRC, not national-level anything, just the people who show up on a cold Saturday morning because they love it. It’s coming together in a way that genuinely surprises me, and I can’t wait to share it. Whether it snowballs beyond that little community or stays exactly where it belongs — either outcome is fine by me. Watch this space. The point is: the muscle-I’m-not-flexing argument has a real counterargument. Some muscles you were never going to flex. The question isn’t whether AI is letting you skip work you would have done — it’s whether it’s letting you skip work you needed to do.
And here’s the nuance I don’t see in any of the research papers: even in my AI-assisted building, I’ve been forced to pick up real architecture and structural knowledge. Not syntax. Not line-by-line debugging. But the big stuff — how systems talk to each other, where failure points live, what good design looks like from ten thousand feet. You can’t direct an AI agent well without developing some of that understanding. Which means the cognitive offloading isn’t total, even when you want it to be.
But. And this is a big but.
Prompt engineering is a critical thinking skill. Full stop. I’ve said this before and I’ll keep saying it: it doesn’t matter what part of AI you’re using — better prompts produce better output, period. And in development specifically, this is where the critical thinking stakes get very concrete very fast.
Coding agents, once they get confused on a complex problem, will loop. They’ll circle the same approach, restate the same error in different words, try the same fix from a slightly different angle, and burn your credits doing it. I watch my agents do this regularly on anything sufficiently complex. And the only thing that breaks the loop is a human stepping back, recognizing the pattern, and redirecting — sometimes firmly. “Stop. What you’re doing isn’t working. Let’s look at this from a completely different angle. Forget your last three approaches.” That’s not a technical skill. That’s critical thinking applied to a system that can’t apply it to itself. I’m considering if I can prompt engineer or persona engineer my manager agents to do this for me – I think I can.
Without that ability — without the instinct to recognize when a process has become circular and needs a reset — you will spend days in a rat hole that an AI dug and then helpfully continued digging, deeper and deeper, while confidently assuring you it was making progress. That costs time. It costs credits. And it produces nothing. The human in the loop isn’t optional. The thinking human in the loop definitely isn’t.
We know AI systems confabulate. We know they produce confidently wrong answers with the same smooth delivery as correct ones. We know that the instinct to critically evaluate that output is being trained out of us by the very experience of using these systems. And yet we are deploying them everywhere, for everything, with very little friction between “AI said it” and “I believe it.”
The liability exposure for organizations letting AI-generated answers flow into real decisions without a critical review layer is real. That’s a separate post. But it’s coming.
Bibliographies Were Never Just About Citing Sources
Here’s something I’ve been thinking about that I don’t see anyone talking about enough.
When I was in high school, every essay came with a requirement: a bibliography. A list of every source you referenced. You had to name the book, the author, the edition, the page number. You had to go find those sources, read them, decide if they supported your argument, and then show your work. It felt like busywork at the time. It was not busywork.
That process — finding a source, evaluating whether it was credible, deciding if it actually said what you thought it said, and then correctly attributing it — was a critical thinking exercise disguised as an administrative task. It forced you to triangulate. It forced you to confront the question of authority: who said this, why should I believe them, and does this actually back my claim? It’s almost exactly the same instinct I try to apply when I include authoritative links in these posts. Same idea, different decade.
Now think about what a student does today. They open a chat window, describe the essay topic, and receive five hundred words of confident, grammatically flawless, internally consistent text — with plausible-sounding “sources” that may or may not actually exist in the form described, or at all. The bibliography is still there. The critical thinking that produced it isn’t.
And here’s the part that should bother everyone: plagiarism detection is now effectively broken. For decades, academia fought academic dishonesty with tools like Turnitin — databases that compared submitted work against indexed text to find copied passages. That arms race had a clear enough battleground: human-written text is distinctive, and copying leaves fingerprints. LLM-generated text leaves no fingerprints, because it isn’t copied from anywhere. It’s synthesized. Every output is technically original at the token level even when it’s conceptually hollow.
Worse — even the best AI-detection tools are increasingly unreliable. A student who wants to defeat a detector doesn’t even need to be sophisticated: rephrase a few sentences, shift the tone slightly, change some word choices, regenerate a paragraph. Most detection systems fold. The cat-and-mouse game that took decades to play out with traditional plagiarism has been compressed into a matter of months, and the mouse is currently winning by a country mile.
This isn’t just an academic integrity problem. It’s a signal of something deeper. The reason we required bibliographies and cited sources wasn’t really about catching cheaters. It was about teaching people to build a position on evidence — to understand that an argument is only as strong as the ground truth it stands on. Strip that requirement of its teeth, and you strip the lesson along with it. What you’re left with is a generation that can produce the form of a well-reasoned argument without any of the cognitive infrastructure behind it.
That’s not a skill deficit. That’s an infrastructure failure. And it’s going to compound.
The Education System Has to Step Up. Like, Yesterday.
I’ve been circling this for a while in this series. I said in the Drug Dealers post that broad access to AI matters for humanity. I said in Falling Behind that the pace is creating real anxiety. But I haven’t said this clearly enough yet: the education system has a bigger job right now than it has had in decades, and I’m not sure it knows it yet.
College Board surveys from 2024–2025 found that teachers overwhelmingly worry AI will impede students’ critical thinking development — while a meaningful share of students aren’t even sure what the problem is. That gap is the problem. Students who have grown up with the answer-at-the-top-of-the-search-results experience don’t have a felt sense of what they’re missing. You can’t miss a muscle you never developed.
Even the White House addressed this in April 2025, calling for AI literacy as a foundational skill from kindergarten through post-secondary. That’s good. But I’d argue AI literacy and critical thinking aren’t the same thing — and we need both. Knowing how to use AI well requires knowing how to question AI well, and that’s a critical thinking problem at its core.
Here’s my genuine take: we should be teaching a standalone critical thinking course starting in high school. Not embedded in English. Not a module inside social studies. A real, credited course — call it nothing but Critical Thinking — that teaches source evaluation, logical fallacies, argument structure, identifying confabulation (human and artificial), and the mechanics of how to form a well-reasoned position when everything around you is trying to form it for you. Maybe that sounds old-fashioned. I’d argue it sounds exactly like what 2026 requires.
The Throttle Metaphor (You Knew It Was Coming)
There’s a point in a rally stage — I’ve been there — where you’re approaching a hairpin and the temptation is to keep the throttle buried because it feels like commitment, like speed, like forward motion. The car feels alive. The numbers feel good. But the driver who survives the stage is the one who knows when to lift — who has the situational awareness to recognize that momentum without judgment is just a faster crash.
AI convenience is the throttle. And right now, most of us — me included, I will be honest — are keeping it buried because it feels amazing. Output is flying. Time is compressing. The machine is singing. I wrote about this in the Acceleration Trap post.
But critical thinking is the lift point. It’s not weakness. It’s not inefficiency. It’s the thing that keeps you on the road. And we are, collectively, conditioning ourselves to be worse at it, faster than we’ve ever conditioned ourselves out of any skill before.
The answer isn’t to stop using AI. That ship has sailed. The answer is to be deliberate — almost aggressive — about preserving and rebuilding the habits of skepticism, source-checking, and independent reasoning that convenience wants to replace. Use AI to do the work. Use your brain to decide if the work is right.
The Bottom Line
We laughed at the “Is Google Making Us Stupid?” debate in 2008 and moved on. We mostly got it wrong — not because Google didn’t change cognition, but because the critical layer was still there in the search results. You still had to pick.
That layer is eroding. Fast. A Frontiers in Education study from late 2025 found that students who over-rely on AI for answers show “substantial declines in analytical reasoning capabilities” — and that younger participants are the most vulnerable. This isn’t speculation. It’s in the data.
We are building faster on a foundation that may be getting softer. And the confabulating, confident, extremely smooth AI systems we’ve all become dependent on are both the cause and the reason we’re not noticing.
I don’t have the five-point fix. I never do in this series — you all know that by now. But I do think it starts with admitting the problem out loud. Demanding that our schools, our workplaces, and maybe ourselves take critical thinking seriously again — not as a soft skill, not as a nice-to-have, but as the fundamental human capability that everything else depends on.
Because here’s the thing: AI can confabulate. Humans can too. The only real defense against both is a mind that was trained to ask why before it asks what.
Your turn: Are you still questioning AI answers, or have you found yourself just… trusting them? Did you used to page through search results and pick your sources — and does that feel quaint now? And teachers or parents reading this — do you think a standalone critical thinking course in high school is worth fighting for? Drop it in the comments. I read every one, even if people are not commenting – I’m questioning, did you make it this far? Was it “TLDR” it’s ok, this is how I dump my raw thoughts on random junk – maybe i’ll get back to talking about actually “tech” stuff one day…
(Catch up on the series if you’re new: Falling Behind → AI Is Underpriced → Copilot Price Jump → Drug Dealers → The Acceleration Trap → Time Back → and now this one.)