Stealing Part of a Production Language Model

2024/03/11 by Nicholas Carlini, Carlini, Nicholas, Daniel Paleka +26 · 11 voices · 26 citations
Computer Science · #Natural Language Processing Techniques #cs.CR

paper · pdf · doi:10.48550/arxiv.2403.06634

openalex publication_date 2024/03/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT or Google's PaLM-2. Specifically, our attack recovers the embedding projection layer (up to symmetries) of a transformer model, given typical API access. For under $20 USD, our attack extracts the entire projection matrix of OpenAI's Ada and Babbage language models. We thereby confirm, for the first time, that these black-box models have a hidden dimension of 1024 and 2048, respectively. We also recover the exact hidden dimension size of the gpt-3.5-turbo model, and estimate it would cost under 2,000 in queries to recover the entire projection matrix. We conclude with potential defenses and mitigations, and discuss the implications of possible future work that could extend our attack.

Cited by

• CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation
• Leaky Language Models: Stealing Architecture and Inference Optimizations via Per-Token Timing
• Tracing LLM Behavior to the Training Data with Empirical Next-Token Distributions
• Every Language Model Has a Forgery-Resistant Signature
• LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users
• Harnessing the Universal Geometry of Embeddings
• Key-Conditioned Orthonormal Transform Gating (K-OTG): Multi-Key Access Control with Hidden-State Scrambling for LoRA-Tuned Models
• Provably Extracting the Features from a General Superposition
• Provably Learning from Modern Language Models via Low Logit Rank
• Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs
• CryptoTensors: A Light-Weight Large Language Model File Format for Highly-Secure Model Distribution
• Log Probability Tracking of LLM APIs
• Label Forensics: Interpreting Hard Labels in Black-Box Text Classifier
• Data Augmentation Techniques to Reverse-Engineer Neural Network Weights from Input-Output Queries
• Expert Selections In MoE Models Reveal (Almost) As Much As Text
• Sequences of Logits Reveal the Low Rank Structure of Language Models
• Fingerprinting LLMs via Prompt Injection
• A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
• Clone What You Can't Steal: Black-Box LLM Replication via Logit Leakage and Distillation
• SoK: Large Language Model Copyright Auditing via Fingerprinting
• Invitation Is All You Need! Promptware Attacks Against LLM-Powered Assistants in Production Are Practical and Dangerous
• Group Fairness Meets the Black Box: Enabling Fair Algorithms on Closed LLMs via Post-Processing
• Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation
• Security Analysis of ChatGPT: Threats and Privacy Risks
• "Energon": Unveiling Transformers from GPU Power and Thermal Side-Channels
• ConfGuard: A Simple and Effective Backdoor Detection for Large Language Models

Discussions

• Stealing Part of a Production Language Model [hn, 218 points, 51 comments]
• Stealing Part of a Production Language Model (2024) [lobsters, 6 points, 0 comments]
• LLMのAPIコールだけでモデルの最終層をシステムから盗み出す実験らしい。ワロタ arxiv.org/abs/2403.06634 [bsky, 3 points, 0 comments]
• arxiv.org/pdf/2403.066... I’m sorry what?! 😂 [bsky, 2 points, 2 comments]
• 'We introduce the first model-stealing attack that extracts precise, nontrivial information from black-box production language models like OpenAI's ChatGPT' arxiv.org/abs/2403.06634 #llms #chatgpt [bsky, 1 points, 0 comments]
• Google shows how to look inside the model black box with their paper: "Stealing Part of a Production Language Model". Their model-stealing attack extracts precise, nontrivial information from black-b [bsky, 1 points, 0 comments]
• Stealing Part of a Production Language Model (2024) https://lobste.rs/s/agrmoo #api #math #ai [bsky, 0 points, 0 comments]
• あとで読む arxiv.org/abs/2403.06634 [bsky, 0 points, 0 comments]
• Stealing Part of a Production Language Model https://arxiv.org/abs/2403.06634 [bsky, 0 points, 0 comments]
• arxiv.org/abs/2403.06634 interesting [bsky, 0 points, 0 comments]
• Google wrote a paper on how an attack can reveal details of an LLM architecture, like the unreleased size of internal GPT-4 embeddings arxiv.org/abs/2403.06634 [bsky, 0 points, 0 comments]

Related

• Logits of API-Protected LLMs Leak Proprietary Information
• Information Leakage from Embedding in Large Language Models
• Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Models
• A Survey on Model Extraction Attacks and Defenses for Large Language Models
• Increasing the Cost of Model Extraction with Calibrated Proof of Work