2024/08/01 by Jy-yong Sohn, Dohyun Kwon, Sohn, Jy-yong +5
Engineering · #3D IC and TSV technologies #Advanced Memory and Neural Computing #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)
paper · pdf · doi:10.48550/arxiv.2408.00359
openalex publication_date 2024/08/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Fine-tuning large pre-trained models is a common practice in machine learning applications, yet its mathematical analysis remains largely unexplored. In this paper, we study fine-tuning through the lens of memorization capacity. Our new measure, the Fine-Tuning Capacity (FTC), is defined as the maximum number of samples a neural network can fine-tune, or equivalently, as the minimum number of neurons (m) needed to arbitrarily change N labels among K samples considered in the fine-tuning process. In essence, FTC extends the memorization capacity concept to the fine-tuning scenario. We analyze FTC for the additive fine-tuning scenario where the fine-tuned network is defined as the summation of the frozen pre-trained network f and a neural network g (with m neurons) designed for fine-tuning. When g is a ReLU network with either 2 or 3 layers, we obtain tight upper and lower bounds on FTC; we show that N samples can be fine-tuned with m=Θ(N) neurons for 2-layer networks, and with m=Θ(√(N)) neurons for 3-layer networks, no matter how large K is. Our results recover the known memorization capacity results when N = K as a special case.