Enhanced Recurrent Neural Tangent Kernels for Non-Time-Series Data

2020/12/09 by Sina Alemohammad, Alemohammad, Sina, Randall Balestriero +6
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Computer science #Exploit #FOS: Computer and information sciences #Gaussian Processes and Bayesian Inference #Generalization #Kernel (algebra) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine learning #Mathematics #Neural Networks and Applications #Pooling #Range (aeronautics) #Recurrent neural network #Series (stratigraphy) #Time Series Analysis and Forecasting #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2012.04859

openalex publication_date 2020/12/09 · arxiv created 2021/10/20 · arxiv updated 2021/10/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Kernels derived from deep neural networks (DNNs) in the infinite-width regime provide not only high performance in a range of machine learning tasks but also new theoretical insights into DNN training dynamics and generalization. In this paper, we extend the family of kernels associated with recurrent neural networks (RNNs), which were previously derived only for simple RNNs, to more complex architectures including bidirectional RNNs and RNNs with average pooling. We also develop a fast GPU implementation to exploit the full practical potential of the kernels. Though RNNs are typically only applied to time-series data, we demonstrate that classifiers using RNN-based kernels outperform a range of baseline methods on 90 non-time-series datasets from the UCI data repository.

Citations

• Modeling of strength of high-performance concrete using artificial neural networks
• Knowledge discovery on RFM model using Bernoulli sequence
• Finding Structure in Time
• Dealing with the Effects of Sensor Displacement in Wearable Activity Recognition
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Universality of deep convolutional neural networks
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• SymJAX: symbolic CPU/GPU/TPU programming
• Infinite attention: NNGP and NTK for deep attention networks
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• Kernel-Based Smoothness Analysis of Residual Networks
• Learning and Generalization in RNNs

Related

• The Recurrent Neural Tangent Kernel
• Random Features for the Neural Tangent Kernel
• Learning with Neural Tangent Kernels in Near Input Sparsity Time
• Scaling Neural Tangent Kernels via Sketching and Random Features
• A Simple Algorithm For Scaling Up Kernel Methods