Breaking the Layer Barrier: Remodeling Private Transformer Inference with Hybrid CKKS and MPC

2025/08/27 by Tianshi Xu, Wenjie Lu, Xu, Tianshi +11 · 3 citations
Computer Science · #Cryptography and Data Security #Privacy-Preserving Technologies in Data #Cryptography and Residue Arithmetic

paper · pdf · doi:10.48550/arxiv.2508.19525

Abstract

This paper presents an efficient framework for private Transformer inference that combines Homomorphic Encryption (HE) and Secure Multi-party Computation (MPC) to protect data privacy. Existing methods often leverage HE for linear layers (e.g., matrix multiplications) and MPC for non-linear layers (e.g., Softmax activation functions), but the conversion between HE and MPC introduces significant communication costs. The proposed framework, dubbed BLB, overcomes this by breaking down layers into fine-grained operators and further fusing adjacent linear operators, reducing the need for HE/MPC conversions. To manage the increased ciphertext bit width from the fused linear operators, BLB proposes the first secure conversion protocol between CKKS and MPC and enables CKKS-based computation of the fused operators. Additionally, BLB proposes an efficient matrix multiplication protocol for fused computation in Transformers. Extensive evaluations on BERT-base, BERT-large, and GPT2-base show that BLB achieves a 21× reduction in communication overhead compared to BOLT (S&P'24) and a 2× reduction compared to Bumblebee (NDSS'25), along with latency reductions of 13× and 1.8×, respectively, when leveraging GPU acceleration.

Citations

• Accelerating Private Large Transformers Inference through Fine-grained Collaborative Computation
• Nimbus: Secure and Efficient Two-Party Inference for Transformers
• Breaking the Layer Barrier: Remodeling Private Transformer Inference with Hybrid CKKS and MPC
• SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPC
• PUMA: Secure Inference of LLaMA-7B in Five Minutes
• LLaMA: Open and Efficient Foundation Language Models
• MPCFormer: fast, performant and private Transformer inference with MPC
• Transformers in Medical Imaging: A Survey
• F1: A Fast and Programmable Accelerator for Fully Homomorphic Encryption (Extended Version)
• SIRNN: A Math Library for Secure RNN Inference
• CrypTFlow: Secure TensorFlow Inference
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Gazelle: A Low Latency Framework for Secure Neural Network Inference
• MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• SafeStats: Efficient 2PC Protocols for Data Statistic-Related Functions
• General Techniques for Reducing Key-Switching Overhead in Privacy-Preserving Two-Party Transformer Inference
• Gecko: Fast Private Inference via Secure Public Encoder Offloading

Related

• CipherFormer: Efficient Transformer Private Inference with Low Round Complexity
• Accelerating Private Large Transformers Inference through Fine-grained Collaborative Computation
• Network and Compiler Optimizations for Efficient Linear Algebra Kernels in Private Transformer Inference
• THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
• A Survey on Private Transformer Inference