2017/08/17 by Engin Arslan, Arslan, Engin, Tevfik Kosar +1
Computer Science · #Advanced Data Storage Technologies #Distributed and Parallel Computing Systems #Distributed systems and fault tolerance
paper · pdf · doi:10.48550/arxiv.1708.05425
Obtaining optimal data transfer performance is of utmost importance to\ntoday's data-intensive distributed applications and wide-area data replication\nservices. Doing so necessitates effectively utilizing available network\nbandwidth and resources, yet in practice transfers seldom reach the levels of\nutilization they potentially could. Tuning protocol parameters such as\npipelining, parallelism, and concurrency can significantly increase utilization\nand performance, however determining the best settings for these parameters is\na difficult problem, as network conditions can vary greatly between sites and\nover time. Nevertheless, it is an important problem, since poor tuning can\ncause either under- or over-utilization of network resources and thus degrade\ntransfer performance. In this paper, we present three algorithms for\napplication-level tuning of different protocol parameters for maximizing\ntransfer throughput in wide-area networks. Our algorithms dynamically tune the\nnumber of parallel data streams per file (for large file optimization), the\nlevel of control channel pipelining (for small file optimization), and the\nnumber of concurrent file transfers to increase I/O throughput (a technique\nuseful for all types of files). The proposed heuristic algorithms improve the\ntransfer throughput up to 10x compared to the baseline and 7x compared to the\nstate of the art solutions.\n