AttackPilot: Autonomous Inference Attacks Against ML Services With LLM-Based Agents

2025/11/24 by Yixin Wu, Rui Wen, Wu, Yixin +7
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Security and Verification in Computing #Software System Performance and Reliability

paper · pdf · doi:10.48550/arxiv.2511.19536

openalex publication_date 2025/11/24 · openalex created_date 2025/11/28 · openalex updated_date 2026/07/28

Abstract

Inference attacks have been widely studied and offer a systematic risk assessment of ML services; however, their implementation and the attack parameters for optimal estimation are challenging for non-experts. The emergence of advanced large language models presents a promising yet largely unexplored opportunity to develop autonomous agents as inference attack experts, helping address this challenge. In this paper, we propose AttackPilot, an autonomous agent capable of independently conducting inference attacks without human intervention. We evaluate it on 20 target services. The evaluation shows that our agent, using GPT-4o, achieves a 100.0% task completion rate and near-expert attack performance, with an average token cost of only 0.627 per run. The agent can also be powered by many other representative LLMs and can adaptively optimize its strategy under service constraints. We further perform trace analysis, demonstrating that design choices, such as a multi-agent framework and task-specific action spaces, effectively mitigate errors such as bad plans, inability to follow instructions, task context loss, and hallucinations. We anticipate that such agents could empower non-expert ML service providers, auditors, or regulators to systematically assess the risks of ML services without requiring deep domain expertise.

Citations

• Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm?
• Large Language Model-Based Agents for Software Engineering: A Survey
• The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
• Agentless: Demystifying LLM-based Software Engineering Agents
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• RepairAgent: An Autonomous, LLM-Based Agent for Program Repair
• Maatphor: Automated Variant Analysis for Prompt Injection Attacks
• SoK: Unintended Interactions among Machine Learning Defenses and Risks
• Formalizing and Benchmarking Prompt Injection Attacks and Defenses
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
• Autonomous Tester Agent Benchmark
• Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
• Membership Inference Attacks by Exploiting Loss Trajectory
• Imperceptible Backdoor Attack: From Input Space to Feature Representation
• Enhanced Membership Inference Attacks against Machine Learning Models
• ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models
• Membership Leakage in Label-Only Exposures
• Label-Only Membership Inference Attacks
• An Overview of Privacy in Machine Learning
• Information Leakage in Embedding Models
• Systematic Evaluation of Privacy Risks of Machine Learning Models
• Cryptanalytic Extraction of Neural Network Models
• Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion
• Thieves on Sesame Street! Model Extraction of BERT-based APIs
• High Accuracy and High Fidelity Extraction of Neural Networks
• Overlearning Reveals Sensitive Attributes
• ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models
• Exploiting Unintended Feature Leakage in Collaborative Learning
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Age Progression/Regression by Conditional Adversarial Autoencoder
• Membership Inference Attacks against Machine Learning Models
• Xception: Deep Learning with Depthwise Separable Convolutions
• Towards Evaluating the Robustness of Neural Networks
• Stealing Machine Learning Models via Prediction APIs
• Adversarial examples in the physical world
• Deep Residual Learning for Image Recognition
• Explaining and Harnessing Adversarial Examples
• Deep Learning Face Attributes in the Wild
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

Related

• Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification
• AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
• Construction and Evaluation of LLM-based agents for Semi-Autonomous penetration testing
• Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents