2012/06/10 by Nikzad Babaii Rizvandi, Javid Taheri, Rizvandi, Nikzad Babaii +5
Computer Science · #Cloud Computing and Resource Management #Data Mining Algorithms and Applications #Software System Performance and Reliability #cs.DC #cs.PF
paper · pdf · doi:10.48550/arxiv.1206.2016
6 pages-submitted to The Thirteenth International Conference on Parallel and Distributed Computing, Applications and Technologies(PDCAT-12), Beijing, China
arxiv created 2012/07/27 · arxiv updated 2013/01/30
In this paper, we study the dependency between configuration parameters and network load of fixed-size MapReduce applications in shuffle phase and then propose an analytical method to model this dependency. Our approach consists of three key phases: profiling, modeling, and prediction. In the first stage, an application is run several times with different sets of MapReduce configuration parameters (here number of mappers and number of reducers) to profile the network load of the application in the shuffle phase on a given cluster. Then, the relation between these parameters and the network load is modeled by multivariate linear regression. For evaluation, three applications (WordCount, Exim Mainlog parsing, and TeraSort) are utilized to evaluate our technique on a 4-node MapReduce private cluster.