2019/01/18 by Yifeng Zhang, Zhang, Yifeng, Ka-Ho Chow +3
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Neural and Evolutionary Computing (cs.NE) #cs.LG #cs.NE #stat.ML
paper · pdf · doi:10.48550/arxiv.1903.02082
arxiv created 2019/01/18 · arxiv updated 2019/03/07
Much sequential data exhibits highly non-uniform information distribution. This cannot be correctly modeled by traditional Long Short-Term Memory (LSTM). To address that, recent works have extended LSTM by adding more activations between adjacent inputs. However, the approaches often use a fixed depth, which is at the step of the most information content. This one-size-fits-all worst-case approach is not satisfactory, because when little information is distributed to some steps, shallow structures can achieve faster convergence and consume less computation resource. In this paper, we develop a Depth-Adaptive Long Short-Term Memory (DA-LSTM) architecture, which can dynamically adjust the structure depending on information distribution without prior knowledge. Experimental results on real-world datasets show that DA-LSTM costs much less computation resource and substantially reduce convergence time by 41.78% and 46.01 %, compared with Stacked LSTM and Deep Transition LSTM, respectively.