2025/11/08 by Ilin, Vasily, Peter Sushko, Krishna, Ranjay +1
Computer Science · Materials Science · #62G07 #68T07 #FOS: Computer and information sciences #G.3 #Gaussian Processes and Bayesian Inference #Generative Adversarial Networks and Image Synthesis #I.2.6 #Machine Learning (cs.LG) #Machine Learning in Materials Science
paper · pdf · doi:10.48550/arxiv.2511.05924
openalex publication_date 2025/11/08 · openalex created_date 2025/11/12 · openalex updated_date 2026/07/28
Estimating probability density and its score from samples remains a core problem in generative modeling, Bayesian inference, and kinetic theory. Existing methods are bifurcated: classical kernel density estimators (KDE) generalize across distributions but suffer from the curse of dimensionality, while modern neural score models achieve high precision but require retraining for every target distribution. We introduce DiScoFormer (Density and Score Transformer), a ``train-once, infer-anywhere" equivariant Transformer that maps i.i.d. samples to both density values and score vectors, generalizing across distributions and sample sizes. Analytically, we prove that self-attention can recover normalized KDE, establishing it as a functional generalization of kernel methods; empirically, individual attention heads learn multi-scale, kernel-like behaviors. The model converges faster and achieves higher precision than KDE for density estimation, and provides a high-fidelity plug-in score oracle for score-debiased KDE, Fisher information computation, and Fokker-Planck-type PDEs.