Unsolved Problems in ML Safety

2021/09/28 by Dan Hendrycks, Hendrycks, Dan, Nicholas Carlini +5 · 2 voices · 88 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Business #Computer science #Field (mathematics) #Risk analysis (engineering) #Robustness (evolution) #Software Reliability and Analysis Research #cs.AI #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2109.13916

published in arXiv (Cornell University) (Cornell University) · Position Paper

openalex publication_date 2021/09/28 · arxiv created 2022/06/16 · arxiv updated 2022/06/20 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), reducing inherent model hazards ("Alignment"), and reducing systemic hazards ("Systemic Safety"). Throughout, we clarify each problem's motivation and provide concrete research directions.

Cited by

• Unsafe at any AUC: Unlearned Lessons From Sociotechnical Disasters for Responsible AI
• An approach to systemic risks of AI through the lens of emergence, collective action problems, and externalities
• Breaking the Block: Preserving Data Continuity to Train Superior SAEs for Instruct Models
• FinBench: Time-Gated Calibration and Uncertainty Benchmarking for Agentic Financial Forecasting
• Extracting books from production language models
• Are Large Language Models Sensitive to the Motives Behind Communication?
• Evolution and The Knightian Blindspot of Machine Learning
• Beyond Context: Large Language Models' Failure to Grasp Users' Intent
• Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models
• MicroProbe: Efficient Reliability Assessment for Foundation Models with Minimal Data
• Latent space analysis and generalization to out-of-distribution data
• Catching Contamination Before Generation: Spectral Kill Switches for Agents
• Model-Free Robust Beamforming in Satellite Downlink using Reinforcement Learning
• Ensuring Robustness in ML-enabled Software Systems: A User Survey
• How Well Can Preference Optimization Generalize Under Noisy Feedback?
• A-IPO: Adaptive Intent-driven Preference Optimization
• Public Opinion on the Politics of AI Alignment: Cross-National Evidence on Expectations for AI Moderation From Germany and the United States
• When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Markets
• General Exploratory Bonus for Optimistic Exploration in RLHF
• Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracy
• ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research
• Theoretical Foundations of Representation Learning using Unlabeled Data: Statistics and Optimization
• From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teaming
• DCV-ROOD Evaluation Framework: Dual Cross-Validation for Robust Out-of-Distribution Detection
• Ensemble Debates with Local Large Language Models for AI Alignment
• Shapley Uncertainty in Natural Language Generation
• Last Layer Hamiltonian Monte Carlo
• On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment
• Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreaking
• Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment
• The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Models
• Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safety
• Towards provable probabilistic safety for scalable embodied AI systems
• RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teaming
• Continual-MEGA: A Large-scale Benchmark for Generalizable Continual Anomaly Detection
• Localizing Persona Representations in LLMs
• SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety
• What Is AI Safety? What Do We Want It to Be?
• Tools in the Loop: Quantifying Uncertainty of LLM Question Answering Systems That Use Tools
• Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
• Pave Your Own Path: Graph Gradual Domain Adaptation on Fused Gromov-Wasserstein Geodesics
• What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants
• Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review
• Beyond Semantic Manipulation: Token-Space Attacks on Reward Models
• When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit
• Jailbreak Detection in Clinical Training LLMs Using Feature-Based Predictive Models
• Studying People to Study AI: Expert Perspectives on the Epistemic Fit and Barriers of Human Research in AI Safety & Ethics
• Agentic Nesting: A New Methodology for Existing Enterprise Application Integration and Services
• Adversarial Training of Reward Models
• AI alignment [wikipedia]
• AI safety [wikipedia]

Discussions

• Unsolved Problems in ML Safety [hn, 61 points, 3 comments]
• Unsolved Problems in Machine Learning Safety [hn, 2 points, 0 comments]

Related

• Taxonomy of Machine Learning Safety: A Survey and Primer
• Is the Rush to Machine Learning Jeopardizing Safety? Results of a Survey
• A causal model of safety assurance for machine learning
• From Hazard Identification to Controller Design: Proactive and LLM-Supported Safety Engineering for ML-Powered Systems
• On the Safety of Machine Learning: Cyber-Physical Systems, Decision Sciences, and Data Products