2025/04/04 by Qi Luo, Florian Schäfer, Luo, Qi +1
Computer Science · #Stochastic Gradient Optimization Techniques #Machine Learning and ELM #Generative Adversarial Networks and Image Synthesis
paper · pdf · doi:10.48550/arxiv.2504.03914
We propose an adaptive randomized truncation estimator for Krylov subspace methods that optimizes the trade-off between the solution variance and the computational cost, while remaining unbiased. The estimator solves a constrained optimization problem to compute the truncation probabilities on the fly, with minimal computational overhead. The problem has a closed-form solution when the improvement of the deterministic algorithm satisfies a diminishing returns property. We prove that obtaining the optimal adaptive truncation distribution is impossible in the general case. Without the diminishing return condition, our estimator provides a suboptimal but still unbiased solution. We present experimental results in GP hyperparameter training and competitive physics-informed neural networks problem to demonstrate the effectiveness of our approach.