2015/10/16 by Ben Krause, Krause, Ben
Computer Science · Decision Sciences · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Stock Market Forecasting Methods #Time Series Analysis and Forecasting #cs.LG #cs.NE #stat.ML
paper · pdf · doi:10.48550/arxiv.1510.04953
arxiv created 2015/10/16 · openalex publication_date 2015/10/16 · arxiv updated 2015/10/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recurrent Neural Networks (RNNs) have long been recognized for their potential to model complex time series. However, it remains to be determined what optimization techniques and recurrent architectures can be used to best realize this potential. The experiments presented take a deep look into Hessian free optimization, a powerful second order optimization method that has shown promising results, but still does not enjoy widespread use. This algorithm was used to train to a number of RNN architectures including standard RNNs, long short-term memory, multiplicative RNNs, and stacked RNNs on the task of character prediction. The insights from these experiments led to the creation of a new multiplicative LSTM hybrid architecture that outperformed both LSTM and multiplicative RNNs. When tested on a larger scale, multiplicative LSTM achieved character level modelling results competitive with the state of the art for RNNs using very different methodology.