2024/01/30 by Lianbo Ma, Ma, Lianbo, Yuee Zhou +7 · 2 citations
Computer Science · Neuroscience · #Brain Tumor Detection and Classification #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2401.16760
openalex publication_date 2024/01/30 · openalex created_date 2024/02/01 · openalex updated_date 2026/07/28
Weight quantization is an effective technique to compress deep neural networks for their deployment on edge devices with limited resources. Traditional loss-aware quantization methods commonly use the quantized gradient to replace the full-precision gradient. However, we discover that the gradient error will lead to an unexpected zig-zagging-like issue in the gradient descent learning procedures, where the gradient directions rapidly oscillate or zig-zag, and such issue seriously slows down the model convergence. Accordingly, this paper proposes a one-step forward and backtrack way for loss-aware quantization to get more accurate and stable gradient direction to defy this issue. During the gradient descent learning, a one-step forward search is designed to find the trial gradient of the next-step, which is adopted to adjust the gradient of current step towards the direction of fast convergence. After that, we backtrack the current step to update the full-precision and quantized weights through the current-step gradient and the trial gradient. A series of theoretical analysis and experiments on benchmark deep models have demonstrated the effectiveness and competitiveness of the proposed method, and our method especially outperforms others on the convergence performance.