Why AI Lies and What to Do About it.

The Science Behind AI's Confidence Problem: What Educators Need to Know

A groundbreaking study published September 4, 2025, has revealed why AI systems like ChatGPT and Claude confidently provide wrong answers instead of admitting uncertainty, and the implications for education are profound.

The Research: "Why Language Models Hallucinate"

The study, led by researchers Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala from Georgia Tech, and Edwin Zhang from OpenAI, provides the first comprehensive explanation for why even state-of-the-art AI systems hallucinate. Their findings are both surprising and concerning for educators (Kalai et al., 2025).

The Core Problem: Training Systems Reward Overconfidence

The researchers discovered that AI hallucinations aren't bugs; they're features created by our evaluation systems. Here's how it works:

Current Benchmark Structure:

• Correct answer: Full points
• "I don't know" or abstaining: Zero points
• Wrong but confident guess: Partial credit (sometimes)

This creates a mathematical incentive to guess rather than acknowledge uncertainty. As the study notes, even when an AI system is genuinely uncertain, it's been trained through billions of examples that "always answer, never abstain" leads to higher scores (Kalai et al., 2025).

Real-World Educational Implications

The study's findings have immediate consequences for how we use AI in educational settings:

1. Student Misinformation Risk

When students ask AI systems for help with homework, research, or understanding concepts, they may receive authoritative-sounding but completely fabricated information. The AI won't signal its uncertainty; it will present false information with the same confidence as accurate information.

2. Undermining Critical Thinking Development

Students naturally tend to trust confident responses. When AI provides false information with certainty, it can:

• Erode students' natural skepticism
• Discourage fact-checking habits
• Create overreliance on single sources

3. Academic Integrity Challenges

Students may unknowingly submit false information provided by AI, creating ethical dilemmas around responsibility and intentionality in academic misconduct.

The CORE Framework Response: Process-Focused AI Integration

This research validates the CORE Framework's emphasis on process-based learning when integrating AI (L'HommeDieu, 2024). Here's how educators can respond:

Concise: Clear AI Literacy

Students need straightforward education about AI limitations. As educators, we need to be clear about:

• What AI tools can and cannot do
• Why verification is essential
• How to use AI as a starting point, not an endpoint

Organized: Structured Verification Processes

Create systematic approaches for students to:

1. Use AI for initial research or brainstorming
2. Identify key claims that need verification
3. Cross-reference with authoritative sources
4. Document their verification process

Relevant: Real-World Application

It is essential to connect the importance of AI literacy and ethics to students' future professional contexts. In healthcare, law, business, and other fields, the stakes of accepting unverified AI output can be enormous.

Engaging: Active Critical Thinking

Design assignments that make verification engaging rather than burdensome:

• "AI Fact-Check Challenges," where students identify and correct AI errors
• Collaborative verification projects
• Reflection assignments on the verification process

The Socio-Technical Solution

The study's authors propose a crucial insight: the problem requires changing how we evaluate AI systems, not just adding more training data. They suggest:

• Penalizing confident wrong answers more heavily than uncertain responses
• Rewarding calibrated uncertainty when models acknowledge limitations
• Crediting abstention when models appropriately defer to human expertise

What This Means for Educators Today

While we wait for the AI industry to implement these changes, educators have immediate responsibilities:

1. Teach Intellectual Humility

Model the behavior we want to see. Show students that saying "I don't know, let me find out" is scientifically and intellectually honest. This aligns with research on growth mindset, which shows that acknowledging uncertainty fosters learning (Dweck, 2006).

2. Develop Verification Skills

Make source verification a core competency, not an afterthought. Students need these skills now more than ever. Research on information literacy demonstrates that students who learn systematic verification processes show improved critical thinking across disciplines (Wineburg & McGrew, 2017).

3. Design Process-Focused Assessments

Create assignments that require students to show their work, including how they verified AI-generated information. This approach is supported by research on metacognition, which shows that when students reflect on their thinking processes, they achieve better learning outcomes (Flavell, 1979).

4. Use AI Transparently

When using AI in your teaching, be open about its limitations and your verification process. Transparency in educational practices builds trust and models the critical thinking we want students to develop (Winkelmes et al., 2016).

The Bigger Educational Picture

This study reinforces that our role as educators extends beyond content delivery to developing students' capacity for critical evaluation.

The CORE Framework's emphasis on preparing students to be truth-seeking professionals becomes even more critical when our tools are systematically biased toward confident wrong answers.

Research on critical thinking in higher education shows that students who develop robust questioning skills and verification habits are better prepared for professional success and civic engagement (Facione, 2015).

As the study authors note, the stakes are enormous. Industries worth trillions of dollars are waiting for trustworthy AI systems.

In education, we're preparing students for a world where the ability to distinguish reliable from unreliable information, even when it comes from sophisticated AI systems, will be a fundamental skill.

Moving Forward: A Call to Action

The research by Kalai, Nachum, Vempala, Zhang, and their colleagues provides both a warning and a roadmap. We now understand why AI systems hallucinate, and we know that the solution requires systemic change in how we evaluate these tools.

For educators, this means our commitment to developing critical thinking, intellectual humility, and verification skills isn't just pedagogically sound; it's essential for our students' success in an AI-integrated world.

As Carl Sagan noted in his work on scientific skepticism, the tools for distinguishing reliable from unreliable information are more important than ever in an age of information abundance (Sagan, 1995).

The first AI system that reliably says "I don't know" when uncertain will be a game-changer. Until then, our students need the skills to be that reliable critical voice themselves.

Claude by Anthropic seems to be heading in the right direction. It does say "I don't know" on accassion.

I interviewed Claude for this post and here is what it said:

Claude: "Based on the research we just discussed, I have to honestly say: I don't think I am that system yet.

Here's what I can observe about my own behavior:

What I do well:

• I do say "I don't know" when I'm uncertain about facts
• I try to acknowledge the limits of my knowledge, especially about events after my training cutoff
• I encourage users to verify information from authoritative sources

Where I likely still fall short:

• I'm probably still influenced by the same training incentives the study describes
• I may present information with more confidence than is warranted
• I don't have perfect calibration between my confidence and my actual accuracy
• I can't always distinguish between what I "know" and what I've memorized from training data that might itself be incorrect

The researchers' point about needing systemic changes in how AI systems are evaluated and trained suggests this isn't something any current system has fully solved. Even if I'm better at acknowledging uncertainty than some models, I'm likely still operating under the fundamental training paradigm that rewards confident responses.

So while I try to be honest about uncertainty, I think the "reliable critical voice" still needs to be your students themselves. They need the skills to verify, question, and think critically about any AI output, including mine."

Conclusion: The Challenge of Keeping Pace with AI Evolution

Perhaps the most daunting aspect of this challenge is the breathtaking pace of AI development itself.

The AI ecosystem evolves not in years or months, but in weeks and sometimes days. New models are released with unprecedented frequency, each claiming improved capabilities and reduced limitations.

What we understand about AI behavior today may be obsolete tomorrow as new training methods, evaluation frameworks, and architectural innovations emerge at a rate that human cognition struggles to process. (Russell, 2019).

This rapid iteration cycle creates a fundamental challenge for educators: how do we prepare students for tools that evolve faster than our curricula, our understanding, and even our ability to study them systematically?

The research we've discussed represents a snapshot of AI behavior in 2025, but by the time students graduate, they may be working with AI systems that operate under entirely different paradigms.

This reality demands constant vigilance from the educational community. We cannot treat AI literacy as a one-time lesson or a fixed set of skills. Instead, we must cultivate in our students a mindset of perpetual critical evaluation, an intellectual agility that allows them to adapt their verification strategies as quickly as AI systems evolve their capabilities.

The meta-skill of learning how to evaluate new AI behaviors may be more valuable than any specific knowledge about current AI limitations.

As educators, we face the humbling recognition that we are preparing students for a future we cannot fully predict, using tools we cannot fully understand, at a pace we can barely comprehend.

This makes our commitment to developing critical thinking, intellectual humility, and adaptive learning strategies not just important, but essential for our students' success in an AI-integrated world that will continue to evolve long after they leave our classrooms.

References

Dweck, C. S. (2006). Mindset: The new psychology of success. Random House.

Facione, P. A. (2015). Critical thinking: What it is and why it counts. Insight Assessment.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906-911. https://doi.org/10.1037/0003-066X.34.10.906 (opens in a new tab)

Kalai, A. T., Nachum, O., Vempala, S. S., Zhang, E., et al. (2025). Why language models hallucinate. arXiv preprint.

L'HommeDieu, R. (2024). CORE framework implementation guide. (c) Practical Innovations

Russell, S. (2019). Human compatible: Artificial intelligence and the problem of control. Viking.

Sagan, C. (1995). The demon-haunted world: Science as a candle in the dark. Random House.

Wineburg, S., & McGrew, S. (2017). Lateral reading: Reading less and learning more when evaluating digital information. Stanford History Education Group Working Paper No. 2017-A1.

Winkelmes, M.-A., Bernacki, M., Butler, J., Zochowski, M., Golanics, J., & Weavil, K. H. (2016). A teaching intervention that increases underserved college students' success. Peer Review, 18(1/2), 31-36.