2023/08/11 by Andrea Gesmundo, Andréa Gesmundo, Kaitlin Maile +2 · 1 voice · 2 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Architecture #Artificial intelligence #Artificial neural network #Computer science #Deep neural networks #Domain Adaptation and Few-Shot Learning #Engineering #Initialization #Neural Networks and Applications #Programming language #Scratch #Transformation (genetics) #Transformer #cs.LG
paper · pdf · doi:10.48550/arxiv.2308.06103
openalex publication_date 2023/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Training state-of-the-art neural networks requires a high cost in terms of compute and time. Model scale is recognized to be a critical factor to achieve and improve the state-of-the-art. Increasing the scale of a neural network normally requires restarting from scratch by randomly initializing all the parameters of the model, as this implies a change of architecture's parameters that does not allow for a straightforward transfer of knowledge from smaller size models. In this work, we propose six composable transformations to incrementally increase the size of transformer-based neural networks while preserving functionality, allowing to expand the capacity of the model as needed. We provide proof of exact function preservation under minimal initialization constraints for each transformation. The proposed methods may enable efficient training pipelines for larger and more powerful models by progressively expanding the architecture throughout training.