To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

2021/02/19 by Zana Buçinca, Maja Barbara Malaya, Krzysztof Z. Gajos · 2 voices · 44 citations
Computer Science · Decision Sciences · Medicine · #Clinical Reasoning and Diagnostic Skills #Decision-Making and Behavioral Economics #Explainable Artificial Intelligence (XAI) #cs.AI #cs.HC

paper · pdf · doi:10.1145/3449287

arxiv published 2021/02/19 · arxiv updated 2021/02/19 · openalex publication_date 2021/04/13 · openalex created_date 2021/05/10 · openalex updated_date 2026/07/29

Abstract

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.

Citations

• Random effects structure for confirmatory hypothesis testing: Keep it maximal
• To Trust Or Not To Trust A Classifier
• On a Method to Measure Supervised Multiclass Model’s Interpretability: Application to Degradation Diagnosis (Short Paper)
• Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom
• Closing the AI accountability gap

Cited by

• Trust and reliance on AI — An experimental study on the extent and costs of overreliance on AI
• When ignorance induces reliance: The role of epistemic knowledge about generative AI in changing health-related decision-making
• Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
• Exploring the Role of Generative AI in Dementia Resilience Building Activities: Uncovering Opportunities and Challenges
• "I Use ChatGPT to Humanize My Words": Affordances and Risks of ChatGPT to Autistic Users
• Towards AI as Colleagues: Multi-Agent System Improves Structured Professional Ideation
• Perceptions of Sentient AI and Other Digital Minds: Evidence from the AI, Morality, and Sentience (AIMS) Survey
• Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
• From Mass Media and Social Media to AI: A Multilevel Framework for Understanding Trust in Generative AI
• Relief or displacement? How teachers are negotiating generative AI's role in their professional practice
• Presenting Large Language Models as Companions Affects What Mental Capacities People Attribute to Them
• Rams, hounds and white boxes: Investigating human–AI collaboration protocols in medical diagnosis
• Shaping History, Responsibly: Seven Principles to Guide the Design of Archival AI Assistants for Cultural Heritage Collections
• Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research
• AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work
• Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
• The impacts of companion AI on human relationships: risks, benefits, and design considerations
• In search of verifiability: Explanations rarely enable complementary performance in AI‐advised decision making
• From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering
• Dialogues with AI Reduce Beliefs in Misinformation but Build No Lasting Discernment Skills
• The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers
• Ecological Validity Missing in AI-Assisted Clinical Decision Support Research: Why Real-World Context Matters
• The flaws of policies requiring human oversight of government algorithms
• “It Became My Buddy, But I’m Not Afraid to Disagree”: A Multi-Session Study of UX Evaluators Collaborating with Conversational AI Assistants
• Better together? Human oversight as means to achieve fairness in the European AI Act governance
• From Editorial Records to Structured Provenance Information: Documenting Warrant in Knowledge Organization Systems Using Large Language Models
• Empirically derived evaluation requirements for responsible deployments of AI in safety-critical settings
• AI Knows Best? The Paradox of Expertise, AI-Reliance, and Performance in Educational Tutoring Decision-Making Tasks
• AI-assisted teams outperform AI-led teams but not human-only teams in assessing research reproducibility in quantitative social science
• Twenty-four years of empirical research on trust in AI: a bibliometric review of trends, overlooked issues, and future directions
• LLMs in the SOC: An Empirical Study of Human-AI Collaboration in Security Operations Centres
• Enhancing XAI Interpretation through a Reverse Mapping from Insights to Visualizations
• ReviseMate: Exploring Contextual Support for Digesting STEM Paper Reviews
• Wisdom of the Crowd, Without the Crowd: A Socratic LLM for Asynchronous Deliberation on Perspectivist Data
• Never tell me the odds: Investigating pro-hoc explanations in medical decision making
• Helping or Homogenizing? GenAI as a Design Partner to Pre-Service SLPs for Just-in-Time Programming of AAC
• The role of critical thinking on undergraduates' reliance behaviours on generative AI in problem‐solving
• Opportunity Versus Threat Appraisals of AI Aids: The Effect of Appraisal Type on Decision Makers' Effort and Compliance When Using Powerful AI Aids
• AI support in self‐regulated learning: A decade of technological evolution and meta‐analysis
• Learn, Explore and Reflect by Chatting: Understanding the Value of an LLM-Based Voting Advice Application Chatbot
• Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models
• AI and XAI second opinion: the danger of false confirmation in human–AI collaboration
• From Conversation to Orchestration: HCI Challenges and Opportunities in Interactive Multi-Agentic Systems
• Design Principles for Generative AI Applications

Discussions

• arxiv.org/abs/2102.09692 This paper shows that cognitive forcing as an intervention strategy reduces overreliance on AI compared to xAI approaches, though it leads to lower trust and preference among [bsky, 3 points, 0 comments]
• To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making arxiv.org/pdf/2102.09692 [bsky, 0 points, 0 comments]

Related

• Engaging with AI: How Interface Design Shapes Human-AI Collaboration in High-Stakes Decision-Making
• Explanations Can Reduce Overreliance on AI Systems During Decision-Making
• Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
• Utilizing Human Behavior Modeling to Manipulate Explanations in AI-Assisted Decision Making: The Good, the Bad, and the Scary
• Contrastive Explanations That Anticipate Human Misconceptions Can Improve Human Decision-Making Skills