Optimizing Hard Thresholding for Sparse Model Discovery

2025/04/28 by Derek Jollie, Jollie, Derek W., Scott G. McCalla +1
Computer Science · Physics and Astronomy · #Advanced Multi-Objective Optimization Algorithms #Dynamical Systems (math.DS) #FOS: Computer and information sciences #FOS: Mathematics #Gaussian Processes and Bayesian Inference #Machine Learning (cs.LG) #Model Reduction and Neural Networks #Nonlinear system #Numerical Analysis (math.NA) #Optimization and Control (math.OC) #Pattern recognition (psychology) #Population #Process (computing) #Selection (genetic algorithm) #Simulated annealing #Thresholding

paper · pdf · doi:10.48550/arxiv.2504.20256

openalex publication_date 2025/04/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Many model selection algorithms rely on sparse dictionary learning to provide interpretable and physics-based governing equations. The optimization algorithms typically use a hard thresholding process to enforce sparse activations in the model coefficients by removing library elements from consideration. By introducing an annealing scheme that reactivates a fraction of the removed terms with a cooling schedule, we are able to improve the performance of these sparse learning algorithms. We concentrate on two approaches to the optimization, SINDy, and an alternative using hard thresholding pursuit. We see in both cases that annealing can improve model accuracy. The effectiveness of annealing is demonstrated through comparisons on several nonlinear systems pulled from convective flows, excitable systems, and population dynamics. Finally we apply these algorithms to experimental data for projectile motion.

Citations

• ADAM-SINDy: An Efficient Optimization Framework for Parameterized Nonlinear Dynamical System Identification
• Time-Series Forecasting, Knowledge Distillation, and Refinement within a Multimodal PDE Foundation Model
• BINDy -- Bayesian identification of nonlinear dynamics with reversible-jump Markov-chain Monte-Carlo
• Statistical Mechanics of Dynamical System Identification
• A Mathematical Guide to Operator Learning
• Multiphysics discovery with moving boundaries using Ensemble SINDy and Peridynamic Differential Operator
• Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants
• A Review of Data-Driven Discovery for Dynamic Systems
• A Review of Data‐Driven Discovery for Dynamic Systems
• Learning theory for dynamical systems
• On Learning the Dynamical Response of Nonlinear Control Systems with Deep Operator Networks
• Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next
• Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next
• SINDy with Control: A Tutorial
• Physics-informed machine learning
• Physics-informed neural networks (PINNs) for fluid mechanics: A review
• Physics-informed neural networks (PINNs) for fluid mechanics: a review
• Bridging Physics-based and Data-driven modeling for Learning Dynamical Systems
• Fourier Neural Operator for Parametric Partial Differential Equations
• Numerical differentiation of noisy data: A unifying multi-objective optimization framework
• DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
• On the Convergence of the SINDy Algorithm
• PDE-Net: Learning PDEs from Data
• Extracting Sparse High-Dimensional Dynamics from Limited Data
• Exact Recovery of Chaotic Systems from Highly Corrupted Data
• Discovering governing equations from data: Sparse identification of nonlinear dynamical systems
• Deterministic Nonperiodic Flow

Related

• Validation-Free Sparse Learning: A Phase Transition Approach to Feature Selection
• Learning Large Scale Sparse Models
• A unified sparse optimization framework to learn parsimonious physics-informed models from data
• Scalable Sparse Regression for Model Discovery: The Fast Lane to Insight
• EsaCL: Efficient Continual Learning of Sparse Models