Diffusion-Assisted Distillation for Self-Supervised Graph Representation Learning with MLPs

2025/10/05 by Ahn, Seong Jin, Kim, Myoung-Ho
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.04241

Abstract

For large-scale applications, there is growing interest in replacing Graph Neural Networks (GNNs) with lightweight Multi-Layer Perceptrons (MLPs) via knowledge distillation. However, distilling GNNs for self-supervised graph representation learning into MLPs is more challenging. This is because the performance of self-supervised learning is more related to the model's inductive bias than supervised learning. This motivates us to design a new distillation method to bridge a huge capacity gap between GNNs and MLPs in self-supervised graph representation learning. In this paper, we propose Diffusion-Assisted Distillation for Self-supervised Graph representation learning with MLPs (DAD-SGM). The proposed method employs a denoising diffusion model as a teacher assistant to better distill the knowledge from the teacher GNN into the student MLP. This approach enhances the generalizability and robustness of MLPs in self-supervised graph representation learning. Extensive experiments demonstrate that DAD-SGM effectively distills the knowledge of self-supervised GNNs compared to state-of-the-art GNN-to-MLP distillation methods. Our implementation is available at https://github.com/SeongJinAhn/DAD-SGM.

Citations

• Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation
• AMD: Automatic Multi-step Distillation of Large-scale Vision Models
• Noise-Free Score Distillation
• Understanding the Effect of Data Augmentation on Knowledge Distillation
• Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework
• Denoising Diffusion Models for Plug-and-Play Image Restoration
• Diffusion Probabilistic Models for Structured Node Classification
• Linkless Link Prediction via Relational Distillation
• DreamFusion: Text-to-3D using 2D Diffusion
• NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs
• Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
• Diffusion-LM Improves Controllable Text Generation
• Equivariant Diffusion for Molecule Generation in 3D
• Unsupervised Network Embedding Beyond Homophily
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Augmentation-Free Self-Supervised Learning on Graphs
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation
• From Canonical Correlation Analysis to Self-supervised Graph Neural Networks
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Improved Denoising Diffusion Probabilistic Models
• Score-Based Generative Modeling through Stochastic Differential Equations
• Denoising Diffusion Implicit Models
• Denoising Diffusion Probabilistic Models
• Bootstrap your own latent: A new approach to self-supervised Learning
• Data Augmentation for Graph Neural Networks
• Deep Graph Contrastive Representation Learning
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Graph Random Neural Network for Semi-Supervised Learning on Graphs
• Contrastive Representation Distillation
• Similarity-Preserving Knowledge Distillation
• Improved Knowledge Distillation via Teacher Assistant
• Deep Graph Infomax
• Graph Attention Networks
• Variational Graph Auto-Encoders
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• DeepWalk

Related

• Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
• Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework
• On Self-Distilling Graph Neural Network
• Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs
• A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation