Training behavior of deep neural network in frequency domain

2018/07/03 by Zhi‐Qin John Xu, Yaoyu Zhang, Xu, Zhi-Qin John +3 · 25 citations
Computer Science · #62-07 #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Mathematics #I.2.6 #Information Theory (cs.IT) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Neural Networks and Applications #Statistics Theory (math.ST)

paper · pdf · doi:10.48550/arxiv.1807.01251

openalex publication_date 2018/07/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Why deep neural networks (DNNs) capable of overfitting often generalize well in practice is a mystery [#zhang2016understanding]. To find a potential mechanism, we focus on the study of implicit biases underlying the training process of DNNs. In this work, for both real and synthetic datasets, we empirically find that a DNN with common settings first quickly captures the dominant low-frequency components, and then relatively slowly captures the high-frequency ones. We call this phenomenon Frequency Principle (F-Principle). The F-Principle can be observed over DNNs of various structures, activation functions, and training algorithms in our experiments. We also illustrate how the F-Principle help understand the effect of early-stopping as well as the generalization of DNNs. This F-Principle potentially provides insights into a general principle underlying DNN optimization and generalization.

Citations

• Opening the Black Box of Deep Neural Networks via Information
• A Closer Look at Memorization in Deep Networks
• Understanding training and generalization in deep learning by Fourier\n analysis
• Frequency Principle in Deep Learning with General Loss Functions and Its Potential Application
• A type of generalization error induced by initialization in deep neural networks

Cited by

• Neonatal Hypoxic-ischaemic Encephalopathy Classification from the EEG and HRV Signals Using a Conformer based Masked Autoencoder
• Overcoming Spectral Bias via Cross-Attention
• An output scaling layer boosts deep neural networks for multiscale ODE systems
• HoloNet: Toward a Unified Einstein-Maxwell-Dilaton Framework of QCD
• From Tables to Signals: Revealing Spectral Adaptivity in TabPFN
• EPSegFZ: Efficient Point Cloud Semantic Segmentation for Few- and Zero-Shot Scenarios with Language Guidance
• Improved Training Technique for Shortcut Models
• General transformation neural networks: A class of parametrized functions for high-dimensional function approximation
• General Fourier Feature Physics-Informed Extreme Learning Machine (GFF-PIELM) for High-Frequency PDEs
• Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise
• A Mutil-conditional Diffusion Transformer for Versatile Seismic Wave Generation
• Evolution of Concepts in Language Model Pre-Training
• Is the Frequency Principle always valid?
• ZORRO: Zero-Knowledge Robustness and Privacy for Split Learning (Full Version)
• Nonlinear Collaborative Scheme for Deep Neural Networks
• Learning functions through Diffusion Maps
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Iterative Low-rank Network for Hyperspectral Image Denoising
• Frequency-adaptive tensor neural networks for high-dimensional multi-scale problems
• The Spectral Bias of the Deep Image Prior
• Computational Advantages of Multi-Grade Deep Learning: Convergence Analysis and Performance Insights
• Theory of the Frequency Principle for General Deep Neural Networks
• Physics-informed network paradigm with data generation and background noise removal for diverse distributed acoustic sensing applications
• High precision PINNs in unbounded domains: application to singularity formulation in PDEs
• LFR-PINO: A Layered Fourier Reduced Physics-Informed Neural Operator for Parametric PDEs

Related

• Overview frequency principle/spectral bias in deep learning
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Theory of the Frequency Principle for General Deep Neural Networks
• An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural Network
• Frequency Principle in Deep Learning Beyond Gradient-descent-based Training