Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognition

2025/05/29 by Masaharu Kagiyama, Kagiyama, Masaharu, Tsuyoshi Okita +1
Computer Science · Neuroscience · #Activity recognition #Artificial Intelligence (cs.AI) #Classifier (UML) #Convolutional neural network #Distillation #EEG and Brain-Computer Interfaces #Enhanced Data Rates for GSM Evolution #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and ELM #Neural Networks and Reservoir Computing #Performance (cs.PF) #Point (geometry)

paper · pdf · doi:10.48550/arxiv.2505.22985

openalex publication_date 2025/05/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

This paper aims to develop an energy-efficient classifier for time-series data by introducing PatchEchoClassifier, a novel model that leverages a reservoir-based mechanism known as the Echo State Network (ESN). The model is designed for human activity recognition (HAR) using one-dimensional sensor signals and incorporates a tokenizer to extract patch-level representations. To train the model efficiently, we propose a knowledge distillation framework that transfers knowledge from a high-capacity MLP-Mixer teacher to the lightweight reservoir-based student model. Experimental evaluations on multiple HAR datasets demonstrate that our model achieves over 80 percent accuracy while significantly reducing computational cost. Notably, PatchEchoClassifier requires only about one-sixth of the floating point operations (FLOPS) compared to DeepConvLSTM, a widely used convolutional baseline. These results suggest that PatchEchoClassifier is a promising solution for real-time and energy-efficient human activity recognition in edge computing environments.

Citations

• Neuromorphic on-chip reservoir computing with spiking neural network architectures
• CrossKD: Cross-Head Knowledge Distillation for Object Detection
• MLP-Mixer: An all-MLP Architecture for Vision
• Training data-efficient image transformers & distillation through attention
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Language Models are Few-Shot Learners
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• Relational Knowledge Distillation

Related

• An Ontology-Based, Fully Probabilistic, Scalable Method for Human Activity Recognition
• HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?
• Dataiku's Solution to SPHERE's Activity Recognition Challenge
• Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning
• BSDGAN: Balancing Sensor Data Generative Adversarial Networks for Human Activity Recognition