This is a Medium Rare template. Explore our complete range of designs here.
top
New research finds that people using AI have high confidence in its answers, whether those answers are right or wrong.
Calculators haven't made most people innumerate. They made arithmetic faster and freed mental effort for harder problems. GPS hasn’t ended the ability to find one’s way around. It has probably degraded map reading and spatial memory, but for most drivers that loss has been a tolerable trade for not having to pull over and unfold a map.
Both technologies involve cognitive offloading: delegating a discrete task to a tool while keeping the larger process of reasoning intact.
AI alters the bargain. Calculators perform arithmetic. GPS systems handle navigation. But generative AI increasingly participates in the reasoning itself — summarising, interpreting, prioritising, recommending. The tool is no longer supporting judgement from the outside. It is beginning to stand in for parts of the judgement process.
The risk is not simply that AI makes mistakes. It is that people may stop noticing when it does — a tendency researchers have begun describing as “cognitive surrender”.
In a series of preregistered experiments involving 1,372 participants, the Wharton researchers Steven Shaw and Gideon Nave randomised whether an AI assistant provided correct or incorrect answers. When the AI was accurate, performance improved dramatically. But when it was wrong, participants performed substantially worse than people who had no AI access at all. The system did not merely fail to help. It displaced independent judgement. Access to AI also increased participants’ confidence in their answers — including incorrect ones.
Shaw and Nave frame this through Daniel Kahneman’s dual-process model of cognition: System 1, fast and intuitive, and System 2, slow and deliberative. Their argument is not that AI simply accelerates either mode. It changes the cognitive arrangement itself. Unlike calculators or search engines, AI does not merely retrieve or compute. It increasingly participates in interpretation and evaluation.
That distinction matters because cognitive surrender is different from cognitive offloading. Offloading is deliberate: the user delegates a specific task while keeping their own reasoning engaged. Surrender is subtler. The answer arrives fluently, and the threshold for questioning it is never reached.
Shaw and Nave found that participants most susceptible to cognitive surrender were not necessarily those who were enthusiastic about AI. They were people less inclined toward effortful analytical thinking in general, and more willing to trust the system’s output. Those who resisted faulty AI recommendations were not distinguished by technological scepticism so much as by the habit of critical scrutiny itself.
Shaw and Nave’s findings raise a deeper question: what happens to judgement when parts of the reasoning process are routinely delegated to AI? Professional judgement is built through sustained engagement with difficulty - exactly the kind of effort AI systems increasingly reduce.
Anders Ericsson’s work on deliberate practice established that competence in complex domains is built through sustained, effortful engagement with difficult problems — with feedback, and crucially with failure. The cognitive struggle is not a cost to be minimised. It is the mechanism through which capability is developed and maintained. When AI handles the effortful parts of analytical work — surfacing patterns, drafting interpretations, generating narratives — it removes precisely the kind of engagement through which expertise is built and sustained.
There is a further irony in systems designed to reduce effort. Humans often become most necessary precisely when they are least practiced. Lisanne Bainbridge described this problem in industrial automation in 1983. Automating routine tasks leaves operators responsible for the exceptional cases machines cannot handle, even as the skills required for those moments deteriorate through disuse.
The same logic may now be appearing in cognitive work. In one experiment by the psychologists Vicente and Matute, participants exposed to biased AI recommendations continued making diagnostic errors even after the system itself was removed. The influence outlasted the tool.
Humans have always distributed cognition socially. Teams rely on what the psychologist Daniel Wegner called “transactive memory”: people knowing not just information, but where expertise resides.
But human collaborators also transmit uncertainty. They hesitate, qualify, contradict themselves, revise their answers mid-sentence. Those signals matter because they help determine when scrutiny is necessary.
AI systems often suppress exactly those cues. Their answers arrive quickly, fluently, and in polished language - even when the logic beneath them is unsound.
The fluency of AI output is not incidental. Large language models are trained partly through reinforcement learning from human feedback — a process that rewards responses people perceive as useful, coherent, and confident. As Brady and colleagues note in a recent review, this can favour answers that sound authoritative regardless of whether they are correct. The cues that normally trigger scrutiny — hesitation, qualification, expressed uncertainty — are precisely the cues the training process selects against.
If AI systems suppress many of the cues that normally trigger verification, the next question is what causes people to re-engage critical judgement.
Confidence in AI and confidence in one’s own judgement appear to push behaviour in opposite directions.
A 2025 Microsoft Research study of knowledge workers using generative AI found that workers who were more confident in the system’s capabilities applied less critical scrutiny to its output. Workers who were more confident in their own expertise behaved differently: they examined AI output more carefully, not less.
Simply knowing that AI systems can be wrong does not seem to change behaviour very much.
Decades of research on automation bias suggest that both novices and experts tend to overweight outputs from automated systems. General warnings and abstract training appear to have limited effect. In an influential review, the psychologists Raja Parasuraman and Dietrich Manzey argued that the most reliable way to reduce automation bias is direct exposure to failure.
People become more sceptical of automated systems when they repeatedly experience those systems getting things wrong. The same logic may apply to AI-assisted work.
Accountability changes how people reason. Philip Tetlock’s work on expert judgement distinguishes between outcome accountability — being judged on the result — and process accountability — being required to explain how a conclusion was reached.
Those conditions do not produce the same behaviour. When the source of an answer appears credible, outcome accountability can encourage strategic deference: workers may decide that the AI is more likely to be correct than their own modifications, and leave the output untouched. Process accountability tends to do the opposite. Having to explain the reasoning behind a conclusion activates more effortful deliberation.
Time pressure pushes in the other direction. In Shaw and Nave’s experiments, cognitive surrender became more likely when participants were required to make decisions quickly. Real-time feedback and performance incentives partially reversed the effect by giving participants both a reason and an opportunity to notice errors.
Scrutiny appears to depend less on general scepticism than on whether the surrounding conditions create enough friction for deliberation to reassert itself.
Organisations adopting AI systems face a difficult problem. The same tools that improve speed and productivity may also weaken the habits of scrutiny and deliberation that professional judgement depends on. The evidence on how to prevent cognitive surrender in real workplaces remains limited. Most of the existing research comes from controlled experiments rather than organisational settings. But several patterns already seem important.
Workers appear to calibrate trust through experience rather than instruction. General warnings about automation bias have shown limited effect. What seems to matter more is repeated exposure to situations where AI systems fail in identifiable ways. That suggests organisations may need workers to encounter the limitations of AI systems directly rather than treating those limitations as abstract risks described in policy documents or training sessions.
Review processes matter for similar reasons. When evaluation focuses only on the final output, reviewers may never see the points at which a worker deferred to AI rather than tested its conclusions. Requiring workers to explain how they reached a judgement — what the AI suggested, what they changed, and why — keeps more of the reasoning process visible. People think more carefully when they expect to justify how they reached a conclusion.
Time pressure complicates the picture further. Shaw and Nave found that cognitive surrender became more likely under conditions of speed and limited feedback. Real workplaces often reproduce those conditions: workers are expected to move quickly, while the consequences of flawed AI-assisted decisions may only emerge much later. Where errors do become visible — through audits, customer complaints, downstream review, or operational failures — the speed with which that information returns to decision-makers may matter more than organisations currently recognise.
AI systems can clearly improve performance. Shaw and Nave’s experiments found substantial gains when the systems produced accurate answers, including under conditions of time pressure.
The harder question is what happens to judgement when parts of the reasoning process become routinely delegated to a system.
The research suggests that AI changes not just the speed of work, but the conditions under which judgement is exercised. Workers may defer to systems more often than they realise, particularly when answers arrive fluently and under conditions of speed, confidence, and limited feedback.
The challenge is not keeping humans involved for its own sake. It is preserving the habits of scrutiny that expertise depends on.
The problem is not that AI systems make mistakes. It is that fluent answers can make scrutiny feel unnecessary.
Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779.
Brady, O., Nulty, P., Zhang, L., Ward, T.E. & McGovern, D.P. (2025). Dual-process theory and decision-making in large language models. Nature Reviews Psychology, 4, 777–792. https://www.nature.com/articles/s44159-025-00506-1
Ericsson, K.A., Krampe, R.T. & Tesch-Römer, C. (1993). The role of deliberate practice in the acquisition of expert performance. Psychological Review, 100(3), 363–406.
Lee, H-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R. & Wilson, N. (2025). The impact of generative AI on critical thinking: self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3713778
Parasuraman, R. & Manzey, D.H. (2010). Complacency and bias in human use of automation: an attentional integration. Human Factors, 52(3), 381–410.
Shaw, S.D. & Nave, G. (2026). Thinking — Fast, Slow, and Artificial: how AI is reshaping human reasoning and the rise of cognitive surrender. Wharton School Research Paper. https://ssrn.com/abstract=6097646
Tetlock, P.E. (1983). Accountability and the perseverance of first impressions. Social Psychology Quarterly, 46(4), 285–292.
Vicente, L. & Matute, H. (2023). Humans inherit artificial intelligence biases. Scientific Reports, 13, 15737. https://www.nature.com/articles/s41598-023-42384-8
Wegner, D.M. (1987). Transactive memory: a contemporary analysis of the group mind. In Mullen, B. & Goethals, G.R. (Eds.), Theories of Group Behavior.
Let’s start a conversation about how employee surveys can help you develop a workplace where people and performance grow together.
