Quantum Relational Knowledge Distillation

2025/08/18 by Chen-Yu Liu, Kuan‐Cheng Chen, Liu, Chen-Yu +7 · 4 citations
Business, Management and Accounting · Computer Science · #Bayesian Modeling and Causal Inference #Big Data and Business Intelligence #FOS: Physical sciences #Quantum Computing Algorithms and Architecture #Quantum Physics (quant-ph)

paper · pdf · doi:10.48550/arxiv.2508.13054

openalex publication_date 2025/08/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Knowledge distillation (KD) is a widely adopted technique for compressing large models into smaller, more efficient student models that can be deployed on devices with limited computational resources. Among various KD methods, Relational Knowledge Distillation (RKD) improves student performance by aligning relational structures in the feature space, such as pairwise distances and angles. In this work, we propose Quantum Relational Knowledge Distillation (QRKD), which extends RKD by incorporating quantum relational information. Specifically, we map classical features into a Hilbert space, interpret them as quantum states, and compute quantum kernel values to capture richer inter-sample relationships. These quantum-informed relations are then used to guide the distillation process. We evaluate QRKD on both vision and language tasks, including CNNs on MNIST and CIFAR-10, and GPT-2 on WikiText-2, Penn Treebank, and IMDB. Across all benchmarks, QRKD consistently improves student model performance compared to classical RKD. Importantly, both teacher and student models remain classical and deployable on standard hardware, with quantum computation required only during training. This work presents the first demonstration of quantum-enhanced knowledge distillation in a fully classical deployment setting.

Citations

• Quantum Knowledge Distillation for Large Language Models
• Quantum-Train-Based Distributed Multi-Agent Reinforcement Learning
• Programming Variational Quantum Circuits with Quantum-Train Agent
• A Kernel Perspective on Distillation-based Collaborative Learning
• A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning
• Quantum-Trained Convolutional Neural Network for Deepfake Audio Detection
• Quantum-Train with Tensor Network Mapping Model and Distributed Circuit Ansatz
• Federated Quantum-Train with Batched Parameter Generation
• Quantum-Train Long Short-Term Memory: Application on Flood Prediction Problem
• QTRL: Toward Practical Quantum Reinforcement Learning via Quantum-Train
• Quantum-Train: Rethinking Hybrid Quantum-Classical Machine Learning in the Model Compression Perspective
• Multi-GPU-Enabled Hybrid Quantum-Classical Workflow in Quantum-HPC Middleware: Applications in Quantum Simulations
• Training Classical Neural Networks by Quantum Machine Learning
• Rethinking Centered Kernel Alignment in Knowledge Distillation
• Neural quantum kernels: training quantum kernels with quantum neural networks
• Learning Quantum Phase Estimation by Variational Quantum Circuits
• Quantum Federated Learning With Quantum Networks
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• Reinforcement Learning Quantum Local Search
• Practical Quantum Search by Variational Quantum Eigensolver on Noisy Intermediate-scale Quantum Hardware
• Implementation of Trained Factorization Machine Recommendation System on Quantum Annealer
• The Complexity of NISQ
• Exponential concentration in quantum kernel methods
• Knowledge Distillation in Quantum Neural Network using Approximate Synthesis
• Parametrized quantum policies for reinforcement learning
• Supervised quantum machine learning models are kernel methods
• On the learnability of quantum neural networks
• A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
• Fundamental tenis : resep meraih kemenangan / Tony Mottram
• Machine learning the thermodynamic arrow of time
• Variational Quantum Circuits for Deep Reinforcement Learning
• Similarity of Neural Network Representations Revisited
• Relational Knowledge Distillation
• Classification with Quantum Neural Networks on Near Term Processors
• Data-Free Knowledge Distillation for Deep Neural Networks
• Theoretical insights into the optimization landscape of over-parameterized shallow neural networks
• Pointer Sentinel Mixture Models
• Quantum Recommendation Systems
• Distilling the Knowledge in a Neural Network
• An introduction to quantum machine learning
• A fast quantum mechanical algorithm for database search

Cited by

• QKAN-LSTM: Quantum-inspired Kolmogorov-Arnold Long Short-term Memory
• Federated Quantum Kernel Learning for Anomaly Detection in Multivariate IoT Time-Series
• You Only Measure Once: On Designing Single-Shot Quantum Machine Learning Models
• Generative IQP Circuit Learning with Physics-Informed Latent Initialization

Related

• Bridging Classical and Quantum Machine Learning: Knowledge Transfer From Classical to Quantum Neural Networks Using Knowledge Distillation
• Distilling the knowledge with quantum neural networks
• Quantum Knowledge Distillation for Large Language Models
• A Survey on Quantum Machine Learning: Current Trends, Challenges, Opportunities, and the Road Ahead
• Quantum Hamiltonian Embedding of Images for Data Reuploading Classifiers