pychop: Emulating Low-Precision Arithmetic in Numerical Methods and Neural Networks

2025/04/10 by Erin Carson, Carson, Erin, Xinye Chen +1 · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Numerical Analysis (math.NA) #cs.LG #cs.NA #math.NA

paper · pdf · doi:10.48550/arxiv.2504.07835

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Motivated by the growing demand for reduced-precision arithmetic in computational science, we exploit lower-precision emulation in Python---widely regarded as the dominant programming language for numerical analysis and machine learning. Low-precision paradigms have revolutionized deep learning by enabling more efficient computation and reduced memory footprint while maintaining model fidelity. To better enable numerical experimentation with and exploration of reduced-precision computation, we developed the pychop, which supports customizable floating-point formats and a comprehensive set of rounding modes in Python, allowing users to benefit from fast, reduced-precision emulation in numerous applications. pychop also introduces interfaces for array and tensor backends, enabling efficient reduced-precision emulation on GPUs for neural network training and inference with unparalleled flexibility. In this paper, we offer a comprehensive exposition of the design and applications of pychop, establishing it as a foundational tool for advancing mixed-precision algorithms. Furthermore, we present empirical results on reduced-precision emulation for image classification and object detection using published datasets, illustrating the sensitivity of the use of low precision and offering valuable insights into its quantization-aware training and post-quantization impacts. pychop enables in-depth investigations into the effects of numerical precision, facilitates the development of novel hardware accelerators, and integrates seamlessly into existing deep learning workflows.

Citations

• Training with Fewer Bits: Unlocking Edge LLMs Training with Stochastic Rounding
• Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization
• Training and inference of large language models using 8-bit floating point
• Training Transformers with 4-bit Integers
• Performance Characterization of using Quantization for DNN Inference on Edge Devices: Extended Version
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• QPyTorch: A Low-Precision Arithmetic Simulation Framework
• RandAugment: Practical automated data augmentation with a reduced search space
• When Does Label Smoothing Help?
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
• mixup: Beyond Empirical Risk Minimization
• Mixed Precision Training
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Improved Regularization of Convolutional Neural Networks with Cutout
• Feature Pyramid Networks for Object Detection
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Deep Residual Learning for Image Recognition
• Fixed Point Quantization of Deep Convolutional Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Deep Learning with Limited Numerical Precision
• Microsoft COCO: Common Objects in Context
• The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web]
• What every computer scientist should know about floating-point arithmetic

Related

• QPyTorch: A Low-Precision Arithmetic Simulation Framework
• WRPN & Apprentice: Methods for Training and Inference using Low-Precision Numerics
• Deploying Customized Data Representation and Approximate Computing in Machine Learning Applications
• MCTensor: A High-Precision Deep Learning Library with Multi-Component Floating-Point
• PyTorch: An Imperative Style, High-Performance Deep Learning Library