Direct search based on probabilistic descent in reduced spaces

2022/04/04 by Lindon Roberts, Roberts, Lindon, Clément W. Royer +1 · 6 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Engineering · #68W20 #90C56 #90C60 #Advanced Image and Video Retrieval Techniques #Diffusion and Search Dynamics #FOS: Mathematics #Optimization and Control (math.OC) #Robotics and Sensor-Based Localization

paper · pdf · doi:10.48550/arxiv.2204.01275

openalex publication_date 2022/04/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we study a generic direct-search algorithm in which the polling directions are defined using random subspaces. Complexity guarantees for such an approach are derived thanks to probabilistic properties related to both the subspaces and the directions used within these subspaces. Our analysis crucially extends previous deterministic and probabilistic arguments by relaxing the need for directions to be deterministically bounded in norm. As a result, our approach encompasses a wide range of new optimal polling strategies that can be characterized using our subspace and direction properties. By leveraging results on random subspace embeddings and sketching matrices, we show that better complexity bounds are obtained for randomized instances of our framework. A numerical investigation confirms the benefit of randomization, particularly when done in subspaces, when solving problems of moderately large dimension.

Cited by

• Complexity guarantees and polling strategies for Riemannian direct-search methods
• Optimization by Directional Attacks: Solving Problems with Neural Network Surrogates
• A Noise-Aware Scalable Subspace Classical Optimizer for the Quantum Approximate Optimization Algorithm
• Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Training
• CLARSTA: A random subspace trust-region algorithm for convex-constrained derivative-free optimization
• Leveraging Coordinate Momentum in SignSGD and Muon: Memory-Optimized Zero-Order

Related

• Direct search for stochastic optimization in random subspaces with zeroth-, first-, and second-order convergence and expected complexity
• High-Dimensional Optimization in Adaptive Random Subspaces
• Expected complexity analysis of stochastic direct-search
• Randomization can be as helpful as a glimpse of the future in online computation
• Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares