Kernelized Classification in Deep Networks

2020/12/08 by Sadeep Jayasumana, Jayasumana, Sadeep, Srikumar Ramalingam +3
Computer Science · #Domain Adaptation and Few-Shot Learning #Face and Expression Recognition #Machine Learning and Data Classification

paper · pdf · doi:10.48550/arxiv.2012.09607

Abstract

We propose a kernelized classification layer for deep networks. Although conventional deep networks introduce an abundance of nonlinearity for representation (feature) learning, they almost universally use a linear classifier on the learned feature vectors. We advocate a nonlinear classification layer by using the kernel trick on the softmax cross-entropy loss function during training and the scorer function during testing. However, the choice of the kernel remains a challenge. To tackle this, we theoretically show the possibility of optimizing over all possible positive definite kernels applicable to our problem setting. This theory is then used to device a new kernelized classification layer that learns the optimal kernel function for a given problem automatically within the deep network itself. We show the usefulness of the proposed nonlinear classification layer on several datasets and tasks.

Citations

• Metric spaces and positive definite functions
• Gradient-based learning applied to document recognition

Related

• Classifying Network Data with Deep Kernel Machines
• Learning Explicit Deep Representations from Deep Kernel Networks
• Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
• Stacked Kernel Network
• To understand deep learning we need to understand kernel learning