2023/05/13 by Louis Andréoli, Stéphane Chrétien, Andreoli, Louis +5
Computer Science · Engineering · Neuroscience · #Advanced Memory and Neural Computing #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Reservoir Computing #Neural dynamics and brain function
paper · pdf · doi:10.48550/arxiv.2305.07908
openalex publication_date 2023/05/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Hardware implementation of neural network are an essential step to implement next generation efficient and powerful artificial intelligence solutions. Besides the realization of a parallel, efficient and scalable hardware architecture, the optimization of the system's extremely large parameter space with sampling-efficient approaches is essential. Here, we analytically derive the scaling laws for highly efficient Coordinate Descent applied to optimizing the readout layer of a random recurrently connection neural network, a reservoir. We demonstrate that the convergence is exponential and scales linear with the network's number of neurons. Our results perfectly reproduce the convergence and scaling of a large-scale photonic reservoir implemented in a proof-of-concept experiment. Our work therefore provides a solid foundation for such optimization in hardware networks, and identifies future directions that are promising for optimizing convergence speed during learning leveraging measures of a neural network's amplitude statistics and the weight update rule.