Linear attention coupled Fourier neural operator for simulation of three-dimensional turbulence

2022/10/09 by Peng Wenhui, Peng, Wenhui, Zelong Yuan +5 · 6 citations
Computer Science · Physics and Astronomy · #Advanced Image Processing Techniques #FOS: Physical sciences #Fluid Dynamics (physics.flu-dyn) #Image and Signal Denoising Methods #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2210.04259

openalex publication_date 2022/10/09 · openalex created_date 2022/10/12 · openalex updated_date 2026/07/28

Abstract

Modeling three-dimensional (3D) turbulence by neural networks is difficult because 3D turbulence is highly-nonlinear with high degrees of freedom and the corresponding simulation is memory-intensive. Recently, the attention mechanism has been shown as a promising approach to boost the performance of neural networks on turbulence simulation. However, the standard self-attention mechanism uses O(n²) time and space with respect to input dimension n, and such quadratic complexity has become the main bottleneck for attention to be applied on 3D turbulence simulation. In this work, we resolve this issue with the concept of linear attention network. The linear attention approximates the standard attention by adding two linear projections, reducing the overall self-attention complexity from O(n²) to O(n) in both time and space. The linear attention coupled Fourier neural operator (LAFNO) is developed for the simulation of 3D turbulence. Numerical simulations show that the linear attention mechanism provides 40% error reduction at the same level of computational cost, and LAFNO can accurately reconstruct a variety of statistics and instantaneous spatial structures of 3D turbulence. The linear attention method would be helpful for the improvement of neural network models of 3D nonlinear problems involving high-dimensional data in other scientific domains.

Cited by

• Conditional neural field for spatial dimension reduction of turbulence data: a comparison study
• Self-Attention to Operator Learning-based 3D-IC Thermal Simulation
• Differentiable Autoencoding Neural Operator for Interpretable and Integrable Latent Space Modeling
• VideoPDE: Unified Generative PDE Solving via Video Inpainting Diffusion Models
• BSA: Ball Sparse Attention for Large-scale Geometries
• History-Aware Neural Operator: Robust Data-Driven Constitutive Modeling of Path-Dependent Materials

Related

• Fourier neural operators for spatiotemporal dynamics in two-dimensional turbulence
• An Implicit Adaptive Fourier Neural Operator for Long-term Predictions of Three-dimensional Turbulence
• LESnets (Large-Eddy Simulation nets): Physics-informed neural operator for large-eddy simulation of turbulence
• A transformer-based neural operator for large-eddy simulation of turbulence
• A transformer-based neural operator for large-eddy simulation of turbulence