2022/08/21 by Xuran Meng, Jianfeng Yao, Meng, Xuran +3 · 1 citation
Computer Science · #62R07 #Algorithms and Data Compression #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Statistics Theory (math.ST)
paper · pdf · doi:10.48550/arxiv.2208.09897
openalex publication_date 2022/08/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent works have demonstrated a double descent phenomenon in over-parameterized learning. Although this phenomenon has been investigated by recent works, it has not been fully understood in theory. In this paper, we investigate the multiple descent phenomenon in a class of multi-component prediction models. We first consider a ''double random feature model'' (DRFM) concatenating two types of random features, and study the excess risk achieved by the DRFM in ridge regression. We calculate the precise limit of the excess risk under the high dimensional framework where the training sample size, the dimension of data, and the dimension of random features tend to infinity proportionally. Based on the calculation, we further theoretically demonstrate that the risk curves of DRFMs can exhibit triple descent. We then provide a thorough experimental study to verify our theory. At last, we extend our study to the ''multiple random feature model'' (MRFM), and show that MRFMs ensembling K types of random features may exhibit (K+1)-fold descent. Our analysis points out that risk curves with a specific number of descent generally exist in learning multi-component prediction models.