Never Skip a Batch: Dense Learning of Temporal GNNs via Adaptive Pseudo-Supervision

2025/05/18 by Panyshev, Alexander, Dmitry Vinichenko, Oleg Travkin +6
Computer Science · #Advanced Graph Neural Networks #FOS: Computer and information sciences #Innovative Human-Technology Interaction #Machine Learning (cs.LG) #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2505.12526

openalex publication_date 2025/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Temporal graph networks suffer from irregular supervision in realworld dynamic graphs, as most minibatches contain few labeled events. The lack of labels leads to high-variance gradient updates and, consequently, slow wall-clock convergence. To constructively reduce sparsity, our Moving-Averaged Labels (MAL) assigns soft pseudo-targets based on past supervised signals using a running label distribution while leaving the loss and the model architecture unchanged. Thus, supervision gaps are replaced with informative signals independent of a temporal graph model and the message passing or memory components used. Theoretical analysis supports our insight that aggregating historical supervision into moving average targets reduces stochastic gradient variance, yielding faster convergence under mild assumptions. Experimentally, for TGNv2 and DyRepv2 (our modification of DyRep) models, MAL boosts predictive performance, establishing a new SOTA, and improves time-to-accuracy (on average 6x faster to reach the top score) for a common suite of Temporal Graph Benchmark datasets.

Citations

• Enhancing the Expressivity of Temporal Graph Networks through Source-Target Identification
• Temporal Graph Benchmark for Machine Learning on Temporal Graphs
• Towards Better Dynamic Graph Learning: New Architecture and Unified Library
• Do We Really Need Complicated Model Architectures For Temporal Networks?
• Informative Pseudo-Labeling for Graph Neural Networks with Few Labels
• Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
• Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks
• Inductive Representation Learning on Temporal Graphs
• Unifying Graph Convolutional Neural Networks and Label Propagation
• Self-training with Noisy Student improves ImageNet classification
• Self-Training With Noisy Student Improves ImageNet Classification
• Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning
• Optimization Methods for Large-Scale Machine Learning
• Adam: A Method for Stochastic Optimization
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Cumulated gain-based evaluation of IR techniques
• A Stochastic Approximation Method

Related

• TGL: A General Framework for Temporal GNN Training on Billion-Scale Graphs
• On the Generalization Capability of Temporal Graph Learning Algorithms: Theoretical Insights and a Simpler Method
• A Batch-Insensitive Dynamic GNN Approach to Address Temporal Discontinuity in Graph Streams
• VISAGNN: Versatile Staleness-Aware Efficient Training on Large-Scale Graphs
• FLASH: Flexible Learning of Adaptive Sampling from History in Temporal Graph Neural Networks