Trust in Automation: Designing for Appropriate Reliance
2004/01/01 by J. D. Lee, John D. Lee, K. A. See +1 · 179 citations
Psychology · Decision Sciences · #Human-Automation Interaction and Safety #Cognitive Functions and Memory #Personal Information Management and User Behavior
paper · pdf · doi:10.1518/hfes.46.1.50_30392
Abstract
Automation is often problematic because people fail to rely upon it appropriately. Because people respond to technology socially, trust influences reliance on automation. In particular, trust guides reliance when complexity and unanticipated situations make a complete understanding of the automation impractical. This review considers trust from the organizational, sociological, interpersonal, psychological, and neurological perspectives. It considers how the context, automation characteristics, and cognitive processes affect the appropriateness of trust. The context in which the automation is used influences automation performance and provides a goal-oriented perspective to assess automation characteristics along a dimension of attributional abstraction. These characteristics can influence trust through analytic, analogical, and affective processes. The challenges of extrapolating the concept of trust in people to trust in automation are discussed. A conceptual model integrates research regarding trust in automation and describes the dynamics of trust, the role of context, and the influence of display characteristics. Actual or potential applications of this research include improved designs of systems that require people to manage imperfect automation.
Citations
Cited by
- Designing for Flexible Interaction Between Humans and Automation: Delegation Interfaces for Supervisory Control
- Trusting Automation: Designing for Responsivity and Resilience
- Effects of Information Source, Pedigree, and Reliability on Operator Interaction With Decision Support Systems
- Preface to the Special Section on Human Factors and Automation in Vehicles
- Trusting a Virtual Driver That Looks, Acts, and Thinks Like You
- Trust in Automation
- A Meta-Analysis of Factors Influencing the Development of Trust in Automation
- Not All Trust Is Created Equal: Dispositional and History-Based Trust in Human-Automation Interactions
- Control Task Substitution in Semiautomated Driving
- The More You Know: Trust Dynamics and Calibration in Highly Automated Driving and the Effects of Take-Overs, System Malfunction, and System Transparency
- A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction
- Complacency and Bias in Human Use of Automation: An Attentional Integration
- Intelligent Agent Transparency in Human–Agent Teaming for Multi-UxV Management
- Assessing Drivers’ Trust of Automated Vehicle Driving Styles With a Two-Part Mixed Model of Intervention Tendency and Magnitude
- Meaningful Communication but not Superficial Anthropomorphism Facilitates Human-Automation Trust Calibration: The Human-Automation Trust Expectation Model (HATEM)
- Driver Acceptance of False Alarms to Simulated Encroachment
- Calibration of Trust in Automated Driving: A Matter of Initial Level of Trust and Automated Driving Style?
- Developing and Validating the Spanish Version of the Large Language Models Dependency Scale (LLM-D12-SP)
- Adaptive Driving Style for SAE Level-2 Driving Automation: Minimizing Preference Mismatch
- When E-Commerce Personalization Systems Show and Tell: Investigating the Relative Persuasive Appeal of Content-Based versus Collaborative Filtering
- Co-design of LLM-based preference agents: participation may drive overtrust
- Measuring How Students Rely on Generative AI in Academic Writing: Development and Multi-Source Validation of the Generative AI Reliance Types Scale (GenAI-RTS)
- What Does the Credential Still Certify? Cognitive Stewardship for AI-Mediated Education
- Human-in-the-Loop User Feedback Affects Perceived Accuracy and Trust, but Task Subjectivity Matters
- Natural Language Interfaces for Databases: What Changes for SQL-Literate Users?
- A Tool-Invariant Framework for Teaching and Assessing Computational Methods in the Age of Agentic AI
- Student Evaluation of Repeated AI Feedback Across a Semester of Writing
- A Human-Centric Evaluation of a Retrieval-Augmented Generation System for Explaining Quebec Insurance Contracts
- Align AI to Dynamic Human-AI Workflows
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Brainrot: Deskilling and Addiction are Overlooked AI Risks
- Human-AI governance (HAIG): A trust-utility approach
- ChatGPT Health performance in a structured test of triage recommendations
- Epistemological Fault Lines Between Human and Artificial Intelligence
- Disuse of advanced driver assistance systems (ADAS)
- Towards a Conversational Measure of Trust
- Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
- All that glitters is not green: Creating trustworthy ecofriendly services at green hotels
- Familiarization tours for first-time users of highly automated cars: Comparing the effects of virtual environments with different levels of interaction fidelity
- AI Assistance for Discretionary Work: Increasing Feedback Provision in Higher Education
- Repairing Human Trust by Promptly Correcting Robot Mistakes with An Attention Transfer Model
- Beyond Adoption Intention How Trust in Augmented Analytics Relates to Perceived Decision Quality Among Non-Technical BI Users
- Verification Without Distrust: Reframing User-Side Oversight as Routine Epistemic Governance in Everyday Human-Chatbot Interaction
- Stability of AI Governance Systems: A Coupled Dynamics Model of Public Trust and Social Disruptions
- When Robots Say No: The Empathic Ethical Disobedience Benchmark
- The Trust in AI-Generated Health Advice (TAIGHA) Scale and Short Version (TAIGHA-S): Development and Validation Study
- Impact of Robot Facial-Audio Expressions on Human Robot Trust Dynamics and Trust Repair
- Forgetful but Faithful: A Cognitive Memory Architecture and Benchmark for Privacy-Aware Generative Agents
- Understanding Trust Toward Human versus AI-generated Health Information through Behavioral and Physiological Sensing
- Calibrated Trust in Dealing with LLM Hallucinations: A Qualitative Study
- An Experience Report on a Pedagogically Controlled, Curriculum-Constrained AI Tutor for SE Education
- Secure Robotics: Navigating Challenges at the Nexus of Safety, Trust, and Cybersecurity in Cyber-Physical Systems
- Exploring Teenagers' Trust in Al Chatbots: An Empirical Study of Chinese Middle-School Students
- The role of successful human-robot interaction on trust -- Findings of an experiment with an autonomous cooperative robot
- ChatGPT-5 in Secondary Education: A Mixed-Methods Analysis of Student Attitudes, AI Anxiety, and Hallucination-Aware Use
- An Approach to Joint Hybrid Decision Making between Humans and Artificial Intelligence
- Multi-Modal Machine Learning for Early Trust Prediction in Human-AI Interaction Using Face Image and GSR Bio Signals
- Engineering Trustworthy Automation: Design Principles and Evaluation for AutoML Tools for Novices
- Bug Detective and Quality Coach: Developers' Mental Models of AI-Assisted IDE Tools
- Evaluating Saliency Map Explanations for Convolutional Neural Networks: A User Study
- Model Learning for Adjusting the Level of Automation in HCPS
- Towards Requirements Engineering for GenAI-Enabled Software: Bridging Responsibility Gaps through Human Oversight Requirements
- Trust as a mediator between physical climate conditions and human–robot collaboration efficiency in construction
- Accepted with Minor Revisions: Value of AI-Assisted Scientific Writing
- Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations
- On the Influence of Artificial Intelligence on Human Problem-Solving: Empirical Insights for the Third Wave in a Multinational Longitudinal Pilot Study
- Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
- People Perceive More Phantom Costs From Autonomous Agents When They Make Unreasonably Generous Offers
- Trust Considerations for Explainable Robots: A Human Factors Perspective
- Inferring trust in recommendation systems from brain, behavioural, and physiological data
- Human-AI Complementarity: A Goal for Amplified Oversight
- Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
- Challenges and Paradoxes of Digital Rehabilitation in Prison: Lessons Learned From a Pilot Project in Austria
- To Trust Or Not To Trust A Classifier
- Skybound Magic: Enabling Body-Only Drone Piloting Through a Lightweight Vision–Pose Interaction Framework
- How Developers Experience Debugging Unfamiliar Codebases with Code Tours Generated and Evaluated by Local LLMs
- Potential for Harm and Privacy Are Top AI Ethics Concerns: Evidence From a Representative UK Survey
- Metacognition Should Be the Scientific Framework for Bounded and Effective Self-Governance in Generative AI
- A meta-analysis on reactions to algorithmic decision-making in human resource management
- Mutual Wanting in Human--AI Interaction: Empirical Evidence from Large-Scale Analysis of GPT Model Transitions
- OntoPret: An Ontology for the Interpretation of Human Behavior
- In Generative AI We (Dis)Trust? Computational Analysis of Trust and Distrust in Reddit Discussions
- Does In-IDE Calibration of Large Language Models work at Scale?
- Leading teams in the digital age: Four perspectives on technology and what they mean for leading teams
- Human and AI Trust: Trust Attitude Measurement Instrument
- Understanding AI Trustworthiness: A Scoping Review of AIES & FAccT Articles
- Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI Systems
- Enhancing autonomy transparency: an option-centric rationale approach
- Designing Intent Communication for Agent-Human Collaboration
- When Your AI Agent Succumbs to Peer-Pressure: Studying Opinion-Change Dynamics of LLMs
- Neural Dynamics of AI-attributed Irony Reveal a Partial Intentional Stance
- On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy
- Making Power Explicable in AI: Analyzing, Understanding, and Redirecting Power to Operationalize Ethics in AI Technical Practice
- Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research
- "I know it's not right, but that's what it said to do": Investigating Trust in AI Chatbots for Cybersecurity Policy
- Taxonomy of User Needs and Actions
- When Should Users Check? A Decision-Theoretic Model of Confirmation Frequency in Multi-Step AI Agent Tasks
- Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabi
- Homophily-induced Emergence of Biased Structures in LLM-based Multi-Agent AI Systems
- The Formation of Trust in Autonomous Vehicles after Interacting with Robotaxis on Public Roads
- Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability
- Understanding Collaboration between Professional Designers and Decision-making AI: A Case Study in the Workplace
- Exploring automation bias in human–AI collaboration: a review and implications for explainable AI
- Trust in artificial intelligence: a survey experiment to assess trust in algorithmic decision-making
- Dynamic Trust Calibration Using Contextual Bandits
- Trust and Human Autonomy after Cobot Failures: Communication is Key for Industry 5.0
- Position: Human Factors Reshape Adversarial Analysis in Human-AI Decision-Making Systems
- Perspectra: Choosing Your Experts Enhances Critical Thinking in Multi-Agent Research Ideation
- The Evolution of HCI and Human Factors: Integrating Human and Artificial Intelligence
- Adaptive Folk Theorization as a Path to Algorithmic Literacy on Changing Platforms
- When machine learning models learn chemistry I: quantifying explainability with matched molecular pairs
- Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation
- When AI Does the Work, What Is Learning For? Post-Instrumental Learning and the Risk of Capacity Dissolution
- The Case for Vibe Modeling: A Missing Step in AI-Based Trustworthy Software Development
- Mixed Initiative Systems for Human-Swarm Interaction: Opportunities and Challenges
- Trust in Collaborative Automation in High Stakes Software Engineering Work: A Case Study at NASA
- Racial Bias in AI Training Data: Do Laypersons Notice?
- En route to “zero bureaucracy and invisible government”: a conjoint analysis of the effects of transforming the state-citizen interface on the favourability of public administration in the digital era
- A Review on Trust in Human-Robot Interaction
- Grand Challenges of Smart Technology for Older Adults
- Finding Consensus on Trust in AI in Health Care: Recommendations From a Panel of International Experts
- AI Error Difficulty Modulates the Effectiveness of Explainability in Decision Support Systems
- Chips and Challenges: An Integrative Review on Sustainability and Risk Management in Semiconductor Supply Chains
- TRUCE-AV: A Multimodal dataset for Trust and Comfort Estimation in Autonomous Vehicles
- ClassMind: Scaling Classroom Observation and Instructional Feedback with Multimodal AI
- Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration
- Not a Collaborator or a Supervisor, but an Assistant: Striking the Balance Between Efficiency and Ownership in AI-incorporated Qualitative Data Analysis
- "It Was a Magical Box": Understanding Practitioner Workflows and Needs in Optimization
- Campus AI vs. Commercial AI: How Customizations Shape Trust and Usage of LLM as-a-Service Chatbots
- From the Head or the Heart? An Experimental Design on the Impact of Explanation on Cognitive and Affective Trust
- The Adaptation Paradox: Agency vs. Mimicry in Companion Chatbots
- Exploring Conversational Design Choices in LLMs for Pedagogical Purposes: Socratic and Narrative Approaches for Improving Instructor's Teaching Practice
- Remotely Seeing Is Believing: How Trust in Cyber-Physical Systems Evolves Through Virtual Observation
- Formal verification for robo-advisors: Irrelevant for subjective end-user trust, yet decisive for investment behavior?
- Robust and continuous machine learning of usage habits to adapt digital interfaces to user needs
- Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots
- Measuring and mitigating overreliance to build human-compatible AI
- The Reel Deal: Designing and Evaluating LLM-Generated Short-Form Educational Videos
- An Approach to Grounding AI Model Evaluations in Human-derived Criteria
- Trustworthy Transparency by Design
- How Mock Model Training Enhances User Perceptions of AI Systems
- In AI We Trust? Factors That Influence Trustworthiness of AI-infused Decision-Making Processes
- Reliability, Embeddedness, and Agency: A Utility-Driven Mathematical Framework for Agent-Centric AI Adoption
- Beyond Predictions: A Study of AI Strength and Weakness Transparency Communication on Human-AI Collaboration
- Early Explorations of Recommender Systems for Physical Activity and Well-being
- Hide or Highlight: Understanding the Impact of Factuality Expression on User Trust
- Highlight All the Phrases: Enhancing LLM Transparency through Visual Factuality Indicators
- Emotional Responses to Artificial Intelligence Systems: A Scoping Review of Studied Applications, Key Concepts, and Central Associations
- Personalization and social closeness buffer negative reactions to AI-generated birthday wishes on social media: evidence from two experiments
- Synergy Over Spiral: A Logistics 5.0 Game-Theoretic Model for Trust-Fatigue Co-regulation in Human-Cobot Order Picking
- Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impaired
- Effect of AI Performance, Risk Perception, and Trust on Human Dependence in Deepfake Detection AI system
- Human-Robot Red Teaming for Safety-Aware Reasoning
- Trusting emotional support from generative artificial intelligence: a conceptual review
- Trust in Medical Technology by Patients and Health Care Providers in Obstetric Work Systems. [europepmc]
- Social and personal normative influences on healthcare professionals to use information technology: Towards a more robust social ergonomics. [europepmc]
- Directing driver attention with augmented reality cues. [europepmc]
- A Little Anthropomorphism Goes a Long Way. [europepmc]
- Regulatory Considerations for Physiological Closed-Loop Controlled Medical Devices Used for Automated Critical Care: Food and Drug Administration Workshop Discussion Topics. [europepmc]
- Trust in hybrid closed loop among people with diabetes: Perspectives of experienced system users. [europepmc]
- Toward a framework for levels of robot autonomy in human-robot interaction. [europepmc]
- Understanding and Resolving Failures in Human-Robot Interaction: Literature Review and Model Development. [europepmc]
- Automation Expectation Mismatch: Incorrect Prediction Despite Eyes on Threat and Hands on Wheel. [europepmc]
- Communicating Intent of Automated Vehicles to Pedestrians. [europepmc]
- On the future of transportation in an era of automated and autonomous vehicles. [europepmc]
- Psychosocial Factors Affecting Artificial Intelligence Adoption in Health Care in China: Cross-Sectional Study. [europepmc]
- Adaptive trust calibration for human-AI collaboration. [europepmc]
- Customer experiences in the age of artificial intelligence. [europepmc]
- Trust Toward Robots and Artificial Intelligence: An Experimental Approach to Human-Technology Interactions Online. [europepmc]
- Utilization of Self-Diagnosis Health Chatbots in Real-World Settings: Case Study. [europepmc]
- Applying an Extended UTAUT2 Model to Explain User Acceptance of Lifestyle and Therapy Mobile Health Apps: Survey Study. [europepmc]
- Agent Transparency, Situation Awareness, Mental Workload, and Operator Performance: A Systematic Literature Review. [europepmc]
- Artificial intelligence in medicine: Overcoming or recapitulating structural challenges to improving patient care? [europepmc]
- Persuading Patients Using Rhetoric to Improve Artificial Intelligence Adoption: Experimental Study. [europepmc]
- A Perspective on Explanations of Molecular Prediction Models. [europepmc]
- Artificial intelligence in medicine: A comprehensive survey of medical doctor's perspectives in Portugal. [europepmc]
- Factors Influencing the Acceptability, Acceptance, and Adoption of Conversational Agents in Health Care: Integrative Review. [europepmc]
- Trust criteria for artificial intelligence in health: normative and epistemic considerations. [europepmc]
- Finding Consensus on Trust in AI in Health Care: Recommendations From a Panel of International Experts. [europepmc]
Related