Wearing a MASK: Compressed Representations of Variable-Length Sequences\n Using Recurrent Neural Tangent Kernels

2020/10/26 by Sina Alemohammad, Hossein Babaei, Alemohammad, Sina +19 · 3 citations
Computer Science · #Time Series Analysis and Forecasting #Generative Adversarial Networks and Image Synthesis #Image Processing and 3D Reconstruction

paper · pdf · doi:10.48550/arxiv.2010.13975

Abstract

High dimensionality poses many challenges to the use of data, from\nvisualization and interpretation, to prediction and storage for historical\npreservation. Techniques abound to reduce the dimensionality of fixed-length\nsequences, yet these methods rarely generalize to variable-length sequences. To\naddress this gap, we extend existing methods that rely on the use of kernels to\nvariable-length sequences via use of the Recurrent Neural Tangent Kernel\n(RNTK). Since a deep neural network with ReLu activation is a Max-Affine Spline\nOperator (MASO), we dub our approach Max-Affine Spline Kernel (MASK). We\ndemonstrate how MASK can be used to extend principal components analysis (PCA)\nand t-distributed stochastic neighbor embedding (t-SNE) and apply these new\nalgorithms to separate synthetic time series data sampled from second-order\ndifferential equations.\n

Cited by

• W4S4: WaLRUS Meets S4 for Long-Range Sequence Modeling
• WaLRUS: Wavelets for Long-range Representation Using SSMs
• SaFARi: State-Space Models for Frame-Agnostic Representation

Related

• Kernel-Based Approaches for Sequence Modeling: Connections to Neural Methods
• The Recurrent Neural Tangent Kernel
• A Simple Algorithm For Scaling Up Kernel Methods
• Efficient NTK using Dimensionality Reduction
• Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels