2024/07/09 by Alessandro Ingrosso, Rosalba Pacelli, Ingrosso, Alessandro +5 · 4 citations
Computer Science · #Disordered Systems and Neural Networks (cond-mat.dis-nn) #FOS: Physical sciences #Machine Learning and ELM #Neural Networks and Applications #Statistical Mechanics (cond-mat.stat-mech)
paper · pdf · doi:10.48550/arxiv.2407.07168
openalex publication_date 2024/07/09 · openalex created_date 2024/07/13 · openalex updated_date 2026/07/28
Transfer learning (TL) is a well-established machine learning technique to boost the generalization performance on a specific (target) task using information gained from a related (source) task, and it crucially depends on the ability of a network to learn useful features. Leveraging recent analytical progress in the proportional regime of deep learning theory (i.e. the limit where the size of the training set P and the size of the hidden layers N are taken to infinity keeping their ratio α= P/N finite), in this work we develop a novel single-instance Franz-Parisi formalism that yields an effective theory for TL in fully-connected neural networks. Unlike the (lazy-training) infinite-width limit, where TL is ineffective, we demonstrate that in the proportional limit TL occurs due to a renormalized source-target kernel that quantifies their relatedness and determines whether TL is beneficial for generalization.