Optimization by Direct Search: New Perspectives on Some Classical and Modern Methods

2003/01/01 by Tamara G. Kolda, Robert Michael Lewis, Virginia Torczon · 1,653 citations
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Advanced Optimization Algorithms Research #Algorithm #Artificial intelligence #Class (philosophy) #Computer science #Convergence (economics) #Direct methods #Focus (optics) #Generalization #Mathematical optimization #Mathematics #Metaheuristic Optimization Algorithms Research #Optimization problem #Theoretical computer science #Variety (cybernetics)

paper · doi:10.1137/s003614450242889

published in SIAM Review 45(3), 385-482 (Society for Industrial and Applied Mathematics)

openalex publication_date 2003/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/23

Abstract

Abstract. Direct search methods are best known as unconstrained optimization techniques that do not explicitly use derivatives. Direct search methods were formally proposed and widely applied in the 1960s but fell out of favor with the mathematical optimization community by the early 1970s because they lacked coherent mathematical analysis. Nonetheless, users remained loyal to these methods, most of which were easy to program, some of which were reliable. In the past fifteen years, these methods have seen a revival due, in part, to the appearance of mathematical analysis, as well as to interest in parallel and distributed computing. This review begins by briefly summarizing the history of direct search methods and considering the special properties of problems for which they are well suited. Our focus then turns to a broad class of methods for which we provide a unifying framework that lends itself to a variety of convergence results. The underlying principles allow generalization to handle bound constraints and linear constraints. We also discuss extensions to problems with nonlinear constraints.

Citations

• Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization
• On search directions for minimization algorithms
• An Automatic Method for Finding the Greatest or Least Value of a Function
• Multiplier and gradient methods
• Lipschitzian optimization without the Lipschitz constant
• An efficient method for finding the minimum of a function of several variables without calculating derivatives
• A Simplex Method for Function Minimization
• Theory of Positive Linear Dependence

Cited by

• Derivative-free optimization approach for structured symmetric matrices with fixed eigenvalues
• When AI Bends Metal: AI-Assisted Optimization of Design Parameters in Sheet Metal Forming
• Complexity guarantees and polling strategies for Riemannian direct-search methods
• Advancing Magnetic Field Homogeneity in 9.4 T Non-NMR Magnet With Ferromagnetic Shimming
• Convergence of trust-region methods based on probabilistic models
• Implementing the Nelder-Mead simplex algorithm with adaptive parameters
• Multicriteria Optimization and Decision Making: Principles, Algorithms and Case Studies
• A Stochastic Derivative Free Optimization Method with Momentum
• Direct-search methods in the year 2025: Theoretical guarantees and algorithmic paradigms
• Direct search methods for an open problem of optimization in systems and control
• Sobolev Seminorm of Quadratic Functions with Applications to Derivative-Free Optimization
• Equation-free optimal switching policies for bistable reacting systems using coarse time-steppers
• Reinforcement learning in densely recurrent biological networks
• Ray-tracing image simulations of transparent objects with complex shape and inhomogeneous refractive index
• Adaptive direct search algorithms for constrained optimization
• NEUZZ: Efficient Fuzzing with Neural Program Smoothing
• Surrogate-based Optimization using Mutual Information for Computer Experiments (optim-MICE)
• A characterization of positive spanning sets with ties to strongly edge-connected digraphs
• Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithm
• Estimating the Optimal Linear Combination of Biomarkers using\n Spherically Constrained Optimization
• Hebbian Plasticity for Improving Perceptual Decisions
• Recursive Modified Pattern Search on High-dimensional Simplex : A\n Blackbox Optimization Technique
• Modeling an Augmented Lagrangian for Blackbox Constrained Optimization
• A good body is all you need: avoiding catastrophic interference via agent architecture search
• Simple Complexity Analysis of Simplified Direct Search
• Expected decrease for derivative-free algorithms using random subspaces
• On the Cardinality of Positively Linearly Independent Sets
• The Complexity Of The NP-Class
• Quantum computational chemistry
• Stochastic Subspace Descent Accelerated via Bi-fidelity Line Search
• A General Stochastic Algorithmic Framework for Minimizing Expensive Black Box Objective Functions Based on Surrogate Models and Sensitivity Analysis
• Voronoi-Elitism Genetic Algorithm: A Generic Derivative-Free Routine With Theory and Implementation for Statistical Optimization
• Nuclear energy density optimization
• Maximal Electromagnetic Coupling Between Arbitrary-Shaped Nanotubes
• A Practical Guide to Optimizing Industrial Thermal Spraying through Comparative Multi-Objective Optimization
• Nonlinear Derivative-free Constrained Optimization with a Penalty-Interior Point Method and Direct Search
• Optimization of dynamic mobile robot path planning based on evolutionary methods
• A systematic approach to Kähler moduli stabilisation
• Resource Estimation for Quantum Variational Simulations of the Hubbard Model
• Spatially Selective Artificial-Noise Aided Transmit Optimization for MISO Multi-Eves Secrecy Rate Maximization
• Direct-search methods for decentralized blackbox optimization
• Accurate microRNA Target Prediction Using Detailed Binding Site Accessibility and Machine Learning on Proteomics Data. [europepmc]
• Optimization of microfluidic microsphere-trap arrays. [europepmc]
• JSim, an open-source modeling system for data analysis. [europepmc]
• Mathematical modeling of plant metabolism-from reconstruction to prediction. [europepmc]
• On the convergence and accuracy of the cardiovascular intrinsic frequency method. [europepmc]
• Resolving coiled shapes reveals new reorientation behaviors in C. elegans. [europepmc]
• Modifying response times in the Simon task with transcranial random noise stimulation. [europepmc]
• Intrinsic Frequency Analysis and Fast Algorithms. [europepmc]
• Robust wavenumber and dispersion calibration for Fourier-domain optical coherence tomography. [europepmc]
• Rapid Optimization of External Quantum Efficiency of Thin Film Solar Cells Using Surrogate Modeling of Absorptivity. [europepmc]
• Parameter Inference for Computational Cognitive Models with Approximate Bayesian Computation. [europepmc]
• Decomposing Simon task BOLD activation using a drift-diffusion model framework. [europepmc]
• EM-driven size reduction and multi-criterial optimization of broadband circularly-polarized antennas using pareto front traversing and design extrapolation. [europepmc]
• High-dimensional normalized data profiles for testing derivative-free optimization algorithms. [europepmc]
• Expedited re-design of multi-band passive microwave circuits using orthogonal scaling directions and gradient-based tuning. [europepmc]
• Considering radial basis function neural network for effective solution generation in metaheuristic algorithms. [europepmc]
• Fast machine-learning-enabled size reduction of microwave components using response features. [europepmc]
• Cost-efficient behavioral modeling of antennas by means of global sensitivity analysis and dimensionality reduction. [europepmc]
• A Review on Injection Molding: Conformal Cooling Channels, Modelling, Surrogate Models and Multi-Objective Optimization. [europepmc]
• Optimization of non-smooth functions via differentiable surrogates. [europepmc]
• Multi-objective artificial-intelligence-based parameter tuning of antennas using variable-fidelity machine learning. [europepmc]
• Mitigating Environmental Impact Through the Use of Rice Husk Ash in Sustainable Concrete: Experimental Study, Numerical Modelling, and Optimisation. [europepmc]
• Reinforcement learning in densely recurrent biological networks. [europepmc]
• Intelligent design breaks the trade-off between energy efficiency and water flux in ultrafast seawater desalination. [europepmc]
• Efficient and Robust Crystal Plasticity Parameter Identification via RSM-GA Coupling: Application to AZ31 Magnesium Alloy with Bimodal Non-Basal Texture [europepmc]

Related

• Direct-search methods in the year 2025: Theoretical guarantees and algorithmic paradigms
• Direct-search methods in the year 2025: Theoretical guarantees and algorithmic paradigms
• Adaptive direct search algorithms for constrained optimization
• Direct search methods for an open problem of optimization in systems and control
• Projected-Search Methods for Bound-Constrained Optimization