Frequency-adaptive tensor neural networks for high-dimensional multi-scale problems

2025/08/21 by Jizu Huang, Huang, Jizu, Qiu, Yue +2 · 1 citation
Mathematics · Physics and Astronomy · Computer Science · #Tensor decomposition and applications #Model Reduction and Neural Networks #Generative Adversarial Networks and Image Synthesis

paper · pdf · doi:10.48550/arxiv.2508.15198

Abstract

Tensor neural networks (TNNs) have demonstrated their superiority in solving high-dimensional problems. However, similar to conventional neural networks, TNNs are also influenced by the Frequency Principle, which limits their ability to accurately capture high-frequency features of the solution. In this work, we analyze the training dynamics of TNNs by Fourier analysis and enhance their expressivity for high-dimensional multi-scale problems by incorporating random Fourier features. Leveraging the inherent tensor structure of TNNs, we further propose a novel approach to extract frequency features of high-dimensional functions by performing the Discrete Fourier Transform to one-dimensional component functions. This strategy effectively mitigates the curse of dimensionality. Building on this idea, we propose a frequency-adaptive TNNs algorithm, which significantly improves the ability of TNNs in solving complex multi-scale problems. Extensive numerical experiments are performed to validate the effectiveness and robustness of the proposed frequency-adaptive TNNs algorithm.

Citations

• Frequency-adaptive Multi-scale Deep Neural Networks
• Functional Tensor Decompositions for Physics-Informed Neural Networks
• Respecting causality for training physics-informed neural networks
• Solving High-dimensional Parametric Elliptic Equation Using Tensor Neural Network
• Solving High Dimensional Partial Differential Equations Using Tensor Neural Network and A Posteriori Error Estimators
• Failure-informed adaptive sampling for PINNs
• Tensor Neural Network and Its Numerical Integration
• Bridging Traditional and Machine Learning-based Algorithms for Solving PDEs: The Random Feature Method
• MIONet: Learning multiple-input operators via tensor product
• MIONet: Learning Multiple-Input Operators via Tensor Product
• DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equations
• DAS-PINNs: A deep adaptive sampling method for solving high-dimensional partial differential equations
• Subspace Decomposition based DNN algorithm for elliptic type multi-scale\n PDEs
• Reproducing Activation Function for Deep Learning
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Language Models are Few-Shot Learners
• Multi-scale Deep Neural Networks for Solving High Dimensional PDEs
• DNN Approximation of Nonlinear Finite Element Equations
• A Phase Shift Deep Neural Network for High Frequency Approximation and Wave Problems
• Explicitizing an Implicit Bias of the Frequency Principle in Two-layer Neural Networks
• Understanding training and generalization in deep learning by Fourier\n analysis
• Training behavior of deep neural network in frequency domain
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The Deep Ritz method: A deep learning-based numerical algorithm for solving variational problems
• Attention Is All You Need
• Tensorizing Neural Networks
• Kolmogorov widths and low-rank approximations of parametric elliptic\n PDEs
• Tensor-Sparsity of Solutions to High-Dimensional Elliptic Partial Differential Equations
• ImageNet classification with deep convolutional neural networks
• Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
• Some Mathematical Notes on Three-Mode Factor Analysis
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Overcoming Spectral Bias via Cross-Attention

Related

• Functional tensor train neural network for solving high-dimensional PDEs
• Tensor-Train Networks for Learning Predictive Modeling of Multidimensional Data
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Adaptive Learning of Tensor Network Structures
• Stable Tensor Neural Networks for Rapid Deep Learning