Complacency and Bias in Human Use of Automation: An Attentional Integration
2010/06/01 by Raja Parasuraman, Dietrich H. Manzey, Dietrich Manzey · 65 citations
Psychology · Neuroscience · #Human-Automation Interaction and Safety #Safety Warnings and Signage #Neural and Behavioral Psychology Studies
paper · doi:10.1177/0018720810376055
Abstract
OBJECTIVE: Our aim was to review empirical studies of complacency and bias in human interaction with automated and decision support systems and provide an integrated theoretical model for their explanation. BACKGROUND: Automation-related complacency and automation bias have typically been considered separately and independently. METHODS: Studies on complacency and automation bias were analyzed with respect to the cognitive processes involved. RESULTS: Automation complacency occurs under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention. Automation complacency is found in both naive and expert participants and cannot be overcome with simple practice. Automation bias results in making both omission and commission errors when decision aids are imperfect. Automation bias occurs in both naive and expert participants, cannot be prevented by training or instructions, and can affect decision making in individuals as well as in teams. While automation bias has been conceived of as a special case of decision bias, our analysis suggests that it also depends on attentional processes similar to those involved in automation-related complacency. CONCLUSION: Complacency and automation bias represent different manifestations of overlapping automation-induced phenomena, with attention playing a central role. An integrated model of complacency and automation bias shows that they result from the dynamic interaction of personal, situational, and automation-related characteristics. APPLICATION: The integrated model and attentional synthesis provides a heuristic framework for further research on complacency and automation bias and design options for mitigating such effects in automated and decision support systems.
Citations
Cited by
- A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction
- Keep Your Scanners Peeled
- Bending the Automation Bias Curve: A Study of Human and AI-Based Decision Making in National Security Contexts
- AI-Integrated Scientific Inquiry: A Practice-Centered Vision for Science Education
- What Does the Credential Still Certify? Cognitive Stewardship for AI-Mediated Education
- The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- The Aura in the Machine: Genealogy and the Status of the Work of Art in the Generative Era
- When Not to Automate: A Formal Protocol for Human Preservation in AI-Optimized Organizations
- Artificial intelligence and deskilling in medicine
- Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents
- On the Influence of Artificial Intelligence on Human Problem-Solving: Empirical Insights for the Third Wave in a Multinational Longitudinal Pilot Study
- Opportunity Versus Threat Appraisals of AI Aids: The Effect of Appraisal Type on Decision Makers' Effort and Compliance When Using Powerful AI Aids
- Generative AI and the augmentation of information practices in knowledge work
- Algorithmic Decision-Making and the Control Problem
- The ethical implications of big data in human resource management
- Regulating for trust: Can law establish trust in artificial intelligence?
- Artificial Intelligence and Management: The Automation–Augmentation Paradox
- Toward Guarantees for Clinical Reasoning in Vision Language Models via Formal Verification
- The AI Cognitive Trojan Horse: How Large Language Models May Bypass Human Epistemic Vigilance
- Human Trust in Artificial Intelligence: Review of Empirical Research
- Design Considerations for Human Oversight of AI: Insights from Co-Design Workshops and Work Design Theory
- Discrimination, intelligence artificielle et decisions algorithmiques
- Discrimination, artificial intelligence, and algorithmic decision-making
- Ironies of Automation: Still Unresolved After All These Years
- When Should Users Check? A Decision-Theoretic Model of Confirmation Frequency in Multi-Step AI Agent Tasks
- AI Where It Matters: Where, Why, and How Developers Want AI Support in Daily Work
- Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
- Does using artificial intelligence assistance accelerate skill decay and hinder skill development without performers’ awareness?
- Artificial intelligence, bias and clinical safety
- Exploring AI scribes in Norwegian general practice: a qualitative individual interview study
- Evaluation of Human Factors–Related Risks in AI-Enabled Medical Devices — A Practical Guide
- Open-Source Large Language Models in Education: A Narrative Review of Evidence, Pedagogical Roles, and Learning Outcomes
- The Flaws of Policies Requiring Human Oversight of Government Algorithms
- Machine heuristic: concept explication and development of a measurement scale
- Automation Bias in Large Language Model–Assisted Diagnostic Reasoning among Physicians Trained in AI Literacy — A Randomized Clinical Trial
- Cognitive Offloading
- Stuck on Suggestions: Automation Bias, the Anchoring Effect, and the Factors That Shape Them in Computational Pathology
- Measuring and mitigating overreliance to build human-compatible AI
- Never tell me the odds: Investigating pro-hoc explanations in medical decision making
- Towards Safe and Comfortable Vehicle Control Transitions: A Systematic Review of Takeover Time, Time Budget, and Takeover Performance
- Technology, cognition and error. [europepmc]
- Automation bias and verification complexity: a systematic review. [europepmc]
- Automation bias in electronic prescribing. [europepmc]
- Regulatory Considerations for Physiological Closed-Loop Controlled Medical Devices Used for Automated Critical Care: Food and Drug Administration Workshop Discussion Topics. [europepmc]
- Simple eye-movement feedback during visual search is not helpful. [europepmc]
- Automation Expectation Mismatch: Incorrect Prediction Despite Eyes on Threat and Hands on Wheel. [europepmc]
- Learning From the Slips of Others: Neural Correlates of Trust in Automated Agents. [europepmc]
- Artificial intelligence, bias and clinical safety. [europepmc]
- Automation-Induced Complacency Potential: Development and Validation of a New Scale. [europepmc]
- Individual Differences in Attributes of Trust in Automation: Measurement and Application to System Design. [europepmc]
- Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI. [europepmc]
- Teasing out Artificial Intelligence in Medicine: An Ethical Critique of Artificial Intelligence and Machine Learning in Medicine. [europepmc]
- Opportunities and challenges of artificial intelligence in the medical field: current application, emerging problems, and problem-solving strategies. [europepmc]
- Ethical Issues in Democratizing Digital Phenotypes and Machine Learning in the Next Generation of Digital Health Technologies. [europepmc]
- Data Dentistry: How Data Are Changing Clinical Care and Research. [europepmc]
- Measurement of Trust in Automation: A Narrative Review and Reference Guide. [europepmc]
- Interpretable, not black-box, artificial intelligence should be used for embryo selection. [europepmc]
- Precision dentistry-what it is, where it fails (yet), and how to get there. [europepmc]
- On doing multi-act arithmetic: A multitrait-multimethod approach of performance dimensions in integrated multitasking. [europepmc]
- Pragmatic AI-augmentation in mental healthcare: Key technologies, potential benefits, and real-world challenges and solutions for frontline clinicians. [europepmc]
- Expectations for Artificial Intelligence (AI) in Psychiatry. [europepmc]
- Machine learning-driven clinical decision support system for concept-based searching: a field trial in a Norwegian hospital. [europepmc]
- Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment. [europepmc]
- Developing trustworthy artificial intelligence: insights from research on interpersonal, human-automation, and human-AI trust. [europepmc]
- A Framework for Interpretability in Machine Learning for Medical Imaging. [europepmc]
Related