Tensor Normalization and Full Distribution Training

2021/09/06 by Wolfgang Fuhl, Fuhl, Wolfgang
Computer Science · Physics and Astronomy · #Computational Physics and Python Applications #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Model Reduction and Neural Networks #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2109.02345

arxiv created 2021/09/06 · openalex publication_date 2021/09/06 · arxiv updated 2021/09/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this work, we introduce pixel wise tensor normalization, which is inserted after rectifier linear units and, together with batch normalization, provides a significant improvement in the accuracy of modern deep neural networks. In addition, this work deals with the robustness of networks. We show that the factorized superposition of images from the training set and the reformulation of the multi class problem into a multi-label problem yields significantly more robust networks. The reformulation and the adjustment of the multi class log loss also improves the results compared to the overlay with only one class as label. https://atreus.informatik.uni-tuebingen.de/seafile/d/8e2ab8c3fdd444e1a135/?p=%2FTNandFDT&mode=list

Citations

• Deep Convolutional Ranking for Multilabel Image Annotation
• Adam: A Method for Stochastic Optimization
• On the difficulty of training Recurrent Neural Networks
• Long Short-Term Memory
• Gradient-based learning applied to document recognition
• Bidirectional recurrent neural networks
• Top-k Multiclass SVM
• PupilNet: Convolutional Neural Networks for Robust Pupil Detection
• Wide Residual Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Fast camera focus estimation for gaze-based focus control
• Eye movement simulation and detector creation to reduce laborious parameter adjustments
• Defend Deep Neural Networks Against Adversarial Examples via Fixed and Dynamic Quantized Activation Functions
• Improving Adversarial Robustness via Promoting Ensemble Diversity
• EMPIR: Ensembles of Mixed Precision Deep Networks for Increased Robustness against Adversarial Attacks
• Multi Layer Neural Networks as Replacement for Pooling Operations
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Multiple Layers of Features from Tiny Images

Related

• Faster Neural Network Training with Approximate Tensor Operations
• Weight and Gradient Centralization in Deep Neural Networks
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Frequency Regularization: Restricting Information Redundancy of Convolutional Neural Networks
• Accelerating Training of Deep Neural Networks with a Standardization Loss