Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM

2024/01/22 by Bingbing Li, Li, Bingbing, Geng Yuan +15 · 1 citation
Engineering · Computer Science · #Ferroelectric and Negative Capacitance Devices #Advanced Memory and Neural Computing #Advanced Neural Network Applications

paper · pdf · doi:10.48550/arxiv.2401.11664

Abstract

Resistive Random Access Memory (ReRAM) has emerged as a promising platform for deep neural networks (DNNs) due to its support for parallel in-situ matrix-vector multiplication. However, hardware failures, such as stuck-at-fault defects, can result in significant prediction errors during model inference. While additional crossbars can be used to address these failures, they come with storage overhead and are not efficient in terms of space, energy, and cost. In this paper, we propose a fault protection mechanism that incurs zero space cost. Our approach includes: 1) differentiable structure pruning of rows and columns to reduce model redundancy, 2) weight duplication and voting for robust output, and 3) embedding duplicated most significant bits (MSBs) into the model weight. We evaluate our method on nine tasks of the GLUE benchmark with the BERT model, and experimental results prove its effectiveness.

Cited by

• Weight Transformations in Bit-Sliced Crossbar Arrays for Fault Tolerant Computing-in-Memory: Design Techniques and Evaluation Framework

Related

• Improving DNN Fault Tolerance using Weight Pruning and Differential Crossbar Mapping for ReRAM-based Edge AI
• Online Soft Error Tolerance in ReRAM Crossbars for Deep Learning Accelerators
• NAPER: Fault Protection for Real-Time Resource-Constrained Deep Neural Networks
• Weight Transformations in Bit-Sliced Crossbar Arrays for Fault Tolerant Computing-in-Memory: Design Techniques and Evaluation Framework
• A Low-cost Fault Corrector for Deep Neural Networks through Range Restriction