SBS: Enhancing Parameter-Efficiency of Neural Representations for Neural Networks via Spectral Bias Suppression

2025/09/09 by Qihu Xie, Yuan Li, Xie, Qihu +3
Computer Science · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2509.07373

openalex publication_date 2025/09/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Implicit neural representations have recently been extended to represent convolutional neural network weights via neural representation for neural networks, offering promising parameter compression benefits. However, standard multi-layer perceptrons used in neural representation for neural networks exhibit a pronounced spectral bias, hampering their ability to reconstruct high-frequency details effectively. In this paper, we propose SBS, a parameter-efficient enhancement to neural representation for neural networks that suppresses spectral bias using two techniques: (1) a unidirectional ordering-based smoothing that improves kernel smoothness in the output space, and (2) unidirectional ordering-based smoothing aware random fourier features that adaptively modulate the frequency bandwidth of input encodings based on layer-wise parameter count. Extensive evaluations on various ResNet models with datasets CIFAR-10, CIFAR-100, and ImageNet, demonstrate that SBS achieves significantly better reconstruction accuracy with less parameters compared to SOTA.

Citations

• Neural Metamorphosis
• Towards Croppable Implicit Neural Representations
• Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization
• Diffusion-Based Neural Network Weights Generation
• NeRN -- Learning Neural Representations for Neural Networks
• Single Image Super-Resolution via a Dual Interactive Implicit Neural Network
• Zero-shot Blind Image Denoising via Implicit Neural Representations
• NeRV: Neural Representations for Videos
• Parameter Prediction for Unseen Deep Architectures
• COIN: COmpression with Implicit Neural representations
• Learning Continuous Image Representation with Local Implicit Image Function
• INeRF: Inverting Neural Radiance Fields for Pose Estimation
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Implicit Neural Representations with Periodic Activation Functions
• Local Implicit Grid Representations for 3D Scenes
• Convolutional Occupancy Networks
• Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
• Fine-Grained Analysis of Optimization and Generalization for\n Overparameterized Two-Layer Neural Networks
• DeepSDF: Learning Continuous Signed Distance Functions for Shape\n Representation
• Occupancy Networks: Learning 3D Reconstruction in Function Space
• Learning Implicit Fields for Generative Shape Modeling
• Graph HyperNetworks for Neural Architecture Search
• Generating Neural Networks with Neural Networks
• HyperNetworks
• Deep Residual Learning for Image Recognition
• Multilayer feedforward networks are universal approximators

Related

• Improved Implicit Neural Representation with Fourier Reparameterized Training
• FreSh: Frequency Shifting for Accelerated Neural Representation Learning
• SASNet: Spatially-Adaptive Sinusoidal Networks for INRs
• NeRN -- Learning Neural Representations for Neural Networks
• Random Weight Factorization Improves the Training of Continuous Neural Representations