CAMformer: Associative Memory is All You Need

2025/11/24 by Tergel Molom-Ochir, Benjamin B. Morris, Molom-Ochir, Tergel +18
Computer Science · Engineering · #Advanced Memory and Neural Computing #Advanced Neural Network Applications #FOS: Computer and information sciences #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Network Packet Processing and Optimization

paper · pdf · doi:10.48550/arxiv.2511.19740

openalex publication_date 2025/11/24 · openalex created_date 2025/11/28 · openalex updated_date 2026/07/28

Abstract

Transformers face scalability challenges due to the quadratic cost of attention, which involves dense similarity computations between queries and keys. We propose CAMformer, a novel accelerator that reinterprets attention as an associative memory operation and computes attention scores using a voltage-domain Binary Attention Content Addressable Memory (BA-CAM). This enables constant-time similarity search through analog charge sharing, replacing digital arithmetic with physical similarity sensing. CAMformer integrates hierarchical two-stage top-k filtering, pipelined execution, and high-precision contextualization to achieve both algorithmic accuracy and architectural efficiency. Evaluated on BERT and Vision Transformer workloads, CAMformer achieves over 10x energy efficiency, up to 4x higher throughput, and 6-8x lower area compared to state-of-the-art accelerators--while maintaining near-lossless accuracy.

Citations

• Hamming Attention Distillation: Binarizing Keys and Queries for Efficient Long-Context Transformers
• Gain Cell-Based Analog Content Addressable Memory for Dynamic Associative tasks in AI
• A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
• SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling
• Analog Content-Addressable Memory from Complementary FeFETs
• TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
• ASMCap: An Approximate String Matching Accelerator for Genome Sequence Analysis Based on Capacitive Content Addressable Memory
• On The Computational Complexity of Self-Attention
• QUIDAM: A Framework for Quantization-Aware DNN Accelerator and Model Co-Exploration
• Deep Random Forest with Ferroelectric Analog Content Addressable Memory
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• A Survey of Transformers
• Transformers in Vision: A Survey
• A³: Accelerating Attention Mechanisms in Neural Networks with\n Approximation
• Learning Deep Transformer Models for Machine Translation
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Bit Fusion: Bit-Level Dynamically Composable Architecture for\n Accelerating Deep Neural Networks
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Content-addressable and associative memory: alternatives to the ubiquitous RAM
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• On Difficulties of Attention Factorization through Shared Memory
• BitStopper: An Efficient Transformer Attention Accelerator via Stage-fusion and Early Termination
• EL-Attention: Memory Efficient Lossless Attention for Generation
• C4CAM: A Compiler for CAM-based In-memory Accelerators
• Sparse Attention Acceleration with Synergistic In-Memory Pruning and On-Chip Recomputation