A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties

2025/12/09 by Wang, Jinghao, Zhang, Ping, Yagemann, Carter
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Information and Cyber Security

paper · doi:10.48550/arxiv.2512.08185

openalex publication_date 2025/12/09 · openalex created_date 2025/12/11 · openalex updated_date 2026/07/28

Abstract

Medical Large Language Models (LLMs) are increasingly deployed for clinical decision support across diverse specialties, yet systematic evaluation of their robustness to adversarial misuse and privacy leakage remains inaccessible to most researchers. Existing security benchmarks require GPU clusters, commercial API access, or protected health data -- barriers that limit community participation in this critical research area. We propose a practical, fully reproducible framework for evaluating medical AI security under realistic resource constraints. Our framework design covers multiple medical specialties stratified by clinical risk -- from high-risk domains such as emergency medicine and psychiatry to general practice -- addressing jailbreaking attacks (role-playing, authority impersonation, multi-turn manipulation) and privacy extraction attacks. All evaluation utilizes synthetic patient records requiring no IRB approval. The framework is designed to run entirely on consumer CPU hardware using freely available models, eliminating cost barriers. We present the framework specification including threat models, data generation methodology, evaluation protocols, and scoring rubrics. This proposal establishes a foundation for comparative security assessment of medical-specialist models and defense mechanisms, advancing the broader goal of ensuring safe and trustworthy medical AI systems.

Citations

• MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
• Tree of Attacks: Jailbreaking Black-Box LLMs Automatically
• Scalable Extraction of Training Data from (Production) Language Models
• Jailbreaking Black Box Large Language Models in Twenty Queries
• SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks
• AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
• Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models
• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Med-Flamingo: a Multimodal Medical Few-shot Learner
• Jailbroken: How Does LLM Safety Training Fail?
• Visual Adversarial Examples Jailbreak Aligned Large Language Models
• DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Towards Expert-Level Medical Question Answering with Large Language Models
• GPT-4 Technical Report
• Large Language Models Encode Clinical Knowledge
• Affective Coherence Monitoring for Transformer-Based Language Models
• Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
• Training language models to follow instructions with human feedback
• Extracting Training Data from Large Language Models
• PRAG: Paninian Retrieval-Augmented Generation for Safety-Critical Medical Question Answering
• PubMedQA: A Dataset for Biomedical Research Question Answering
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Adversarial attacks on medical machine learning
• Membership Inference Attacks against Machine Learning Models
• Concrete Problems in AI Safety

Related

• Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare
• Data Poisoning Vulnerabilities Across Healthcare AI Architectures: A Security Threat Analysis
• SoK: Security and Privacy Risks of Healthcare AI
• Explainable Machine Learning-Based Security and Privacy Protection Framework for Internet of Medical Things Systems
• CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs