Partial-Hessian Strategies for Fast Learning of Nonlinear Embeddings

2012/06/18 by Max Vladymyrov, Vladymyrov, Max, Miguel . Carreira-perpi n +1
Computer Science · #Advanced Graph Neural Networks #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.1206.4646

openalex publication_date 2012/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Stochastic neighbor embedding (SNE) and related nonlinear manifold learning algorithms achieve high-quality low-dimensional representations of similarity data, but are notoriously slow to train. We propose a generic formulation of embedding algorithms that includes SNE and other existing algorithms, and study their relation with spectral methods and graph Laplacians. This allows us to define several partial-Hessian optimization strategies, characterize their global and local convergence, and evaluate them empirically. We achieve up to two orders of magnitude speedup over existing training methods with a strategy (which we call the spectral direction) that adds nearly no overhead to the gradient and yet is simple, scalable and applicable to several existing and future embedding algorithms.

Citations

• Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
• Numerical optimization

Related

• Generative Locally Linear Embedding
• An Explicit Nonlinear Mapping for Manifold Learning
• Locally Linear Embedding and its Variants: Tutorial and Survey
• Manifold Learning: The Price of Normalization
• Stochastic Neighbor Embedding with Gaussian and Student-t Distributions: Tutorial and Survey