2016/12/12 by Soheil Hashemi, Hashemi, Soheil, Nicholas Anthony +7 · 1 citation
Computer Science · Engineering · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #CCD and CMOS Imaging Sensors #FOS: Computer and information sciences #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE)
paper · pdf · doi:10.48550/arxiv.1612.03940
openalex publication_date 2016/12/12 · openalex created_date 2022/10/03 · openalex updated_date 2026/07/28
Deep neural networks are gaining in popularity as they are used to generate\nstate-of-the-art results for a variety of computer vision and machine learning\napplications. At the same time, these networks have grown in depth and\ncomplexity in order to solve harder problems. Given the limitations in power\nbudgets dedicated to these networks, the importance of low-power, low-memory\nsolutions has been stressed in recent years. While a large number of dedicated\nhardware using different precisions has recently been proposed, there exists no\ncomprehensive study of different bit precisions and arithmetic in both inputs\nand network parameters. In this work, we address this issue and perform a study\nof different bit-precisions in neural networks (from floating-point to\nfixed-point, powers of two, and binary). In our evaluation, we consider and\nanalyze the effect of precision scaling on both network accuracy and hardware\nmetrics including memory footprint, power and energy consumption, and design\narea. We also investigate training-time methodologies to compensate for the\nreduction in accuracy due to limited bit precision and demonstrate that in most\ncases, precision scaling can deliver significant benefits in design metrics at\nthe cost of very modest decreases in network accuracy. In addition, we propose\nthat a small portion of the benefits achieved when using lower precisions can\nbe forfeited to increase the network size and therefore the accuracy. We\nevaluate our experiments, using three well-recognized networks and datasets to\nshow its generality. We investigate the trade-offs and highlight the benefits\nof using lower precisions in terms of energy and memory footprint.\n