The Differentiation of Pseudo-Inverses and Nonlinear Least Squares Problems Whose Variables Separate
1973/04/01 by Gene H. Golub, G. H. Golub, V. Pereyra +1 · 1,427 citations
Computer Science · Mathematics · #Advanced Optimization Algorithms Research #Combinatorics #Geometry #Inverse #Iterative Methods for Nonlinear Equations #Mathematics #Matrix (chemical analysis) #Matrix Theory and Algorithms #Rank (graph theory)
paper · doi:10.1137/0710036
published in SIAM Journal on Numerical Analysis 10(2), 413-432 (Society for Industrial and Applied Mathematics)
openalex publication_date 1973/04/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/25
Abstract
For given data (ti ,yi ),i = 1, ⋯ ,m, we consider the least squares fit of nonlinear models of the form η (\bf a,\boldsymbol α ;t) = ∑ j = 1n aj φ j (\boldsymbol α ;t), \bf a ∈ Rn , \boldsymbol α ∈ Rk . For this purpose we study the minimization of the nonlinear functional r(\bf a,\boldsymbol α ) = ∑i = 1m ( yi - η ( \bf a,\boldsymbol α ,ti ) )2 . It is shown that by defining the matrix \ \bf Φ (\boldsymbol α )\ i,j = φ j (\boldsymbol α ;ti ), and the modified functional r2 (\boldsymbol α ) = ‖ \bf y - \bf Φ (\boldsymbol α )\bf Φ ^ + (\boldsymbol α )\bf y ‖22 , it is possible to optimize first with respect to the parameters \boldsymbol α , and then to obtain, a posteriors, the optimal parameters \bf a. The matrix \bf Φ ^ + (\boldsymbolα ) is the Moore–Penrose generalized inverse of \bf Φ (\boldsymbolα ). We develop formulas for the Frechet derivative of orthogonal projectors associated with \bf Φ (\boldsymbolα ) and also for \bf Φ ^ + (\boldsymbolα ), under the hypothesis that \bf Φ (\boldsymbolα ) is of constant (though not necessarily full) rank. Detailed algorithms are presented which make extensive use of well-known reliable linear least squares techniques, and numerical results and comparisons are given. These results are generalizations of those of H. D. Scolnik [20] and Guttman, Pereyra and Scolnik [9].
Cited by
- Characterization of the Basin of Convexity for Multi-Snapshot Spike Deconvolution via Variable Projection
- Non-smooth Variable Projection
- Inverse obstacle scattering regularized by the tangent-point energy
- Covariance Estimation for Matrix-variate Data via Fixed-rank Core Covariance Geometry
- ESPRIT-estimation of signal parameters via rotational invariance techniques
- Quotient Manifold Optimization for Spectral Compressed Sensing
- A joint optimization approach to identifying sparse dynamics using least squares kernel collocation
- A Saddle Point Remedy: Power of Variable Elimination in Non-convex Optimization
- Multiple invariance ESPRIT
- Convex Efficient Coding
- Variable Projected Augmented Lagrangian Methods for Generalized Lasso Problems
- Matrix-Free Least Squares Solvers: Values, Gradients, and What to Do With Them
- Learning Multi-Index Models with Hyper-Kernel Ridge Regression
- Omni-QALAS: Optimized Multiparametric Imaging for Simultaneous T1, T2 and Myelin Water Mapping
- Optimization over Geodesics for Exact Principal Geodesic Analysis
- Strong Basin of Attraction for Unmixing Kernels With the Variable Projection Method
- Some comments about measures, Jacobians and Moore-Penrose inverse
- Latent Twins
- Communication-efficient Algorithms for Distributed Stochastic Principal Component Analysis
- Least Squares Rational Approximation
- Algebraic Approach to Ridge-Regularized Mean Squared Error Minimization in Minimal ReLU Neural Network
- Switching strategy based on homotopy continuation for non-regular affine systems with application in induction motor control
- Low-rank matrix completion by Riemannian optimization---extended version
- Variable projection for nonlinear least squares problems
- Prioritized Inverse Kinematics: Nonsmoothness, Trajectory Existence, Task Convergence, Stability
- Hierarchical Maximum Likelihood Estimation for Time-Resolved NMR Data
- Sparse-mode Dynamic Mode Decomposition for Disambiguating Local and Global Structures
- Robustness of DC Power Networks under Weight Control
- Supervised Learning of Sparsity-Promoting Regularizers for Denoising
- On one-sided (B,C)-inverses of arbitrary matrices
- A numerical investigation of matched source waveform inversion applied to acoustic transmission data
- Time-dependent Gaussian basis sets for many-body systems using Rothe's method: A mean-field study
- Models of Heavy-Tailed Mechanistic Universality
- Second-order AAA algorithms for structured data-driven modeling
- Perturbation theory for pseudo-inverses
- Trimmed Constrained Mixed Effects Models: Formulations and Algorithms
- An Exact Penalty Approach for Equality Constrained Optimization over a Convex Set
- A Hybrid Subgradient Method for Nonsmooth Nonconvex Bilevel Optimization
- Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systems
- Physics-informed Temporal Alignment for Auto-regressive PDE Foundation Models
- CNO-driven winds of hot first stars
- Optimal and Adaptive Bayesian Sampling for Non-Linear Parameter Estimation under White Noise
- Fitting Linear Mixed-Effects Models Using lme4
- Orthogonal reparametrization of the Nelson-Siegel-Svensson interest rate curve model: conditioning, diagnostics, and identifiability
- ON ESTIMATING THRESHOLDS IN AUTOREGRESSIVE MODELS
- The ubiquitous Kronecker product
- Geometry of nonlinear least squares with applications to sloppy models and optimization
- Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regime
- Least-squares fitting of circles and ellipses
- Moore–Penrose inverse [wikipedia]
- Generalized separable parameter space techniques for fitting 1K-5K serial compartment models. [europepmc]
- A Constrained Variable Projection Reconstruction Method for Photoacoustic Computed Tomography Without Accurate Knowledge of Transducer Responses. [europepmc]
- Improved approximation of spatial light distribution. [europepmc]
- PyLDM - An open source package for lifetime density analysis of time-resolved spectroscopic data. [europepmc]
- Quantitative magnetic resonance imaging of the upper trapezius muscles - assessment of myofascial trigger points in patients with migraine. [europepmc]
- T2 mapping of the distal sciatic nerve in healthy subjects and patients suffering from lumbar disc herniation with nerve compression. [europepmc]
- A simplified charge projection scheme for long-range electrostatics in ab initio QM/MM calculations. [europepmc]
- A Note on the Connection Between Trek Rules and Separable Nonlinear Least Squares in Linear Structural Equation Models. [europepmc]
- A Graph-Based Framework for Multiscale Modeling of Physiological Transport. [europepmc]
- Statistical learning of protein elastic network from positional covariance matrix. [europepmc]
- Retrospective quantification of clinical abdominal DCE-MRI using pharmacokinetics-informed deep learning: a proof-of-concept study. [europepmc]
- Assessment of training-associated changes of the lumbar back muscle using a multiparametric MRI protocol. [europepmc]
- Time-Dependent Gaussian Basis Sets for Many-Body Systems Using Rothe's Method: A Mean-Field Study. [europepmc]
Related