[M-GI37] Earth and planetary informatics with huge data management

[MGI37-P10] Experimental Evaluation of High-Speed File Transfer over JHPCN

*Praphan Pavarangkoon¹, Ken T. Murata¹, Kazunori Yamamoto¹, Takamichi Mizuhara², Yuya Kagebayashi², Ayahiro Takaki², Kazuya Muranaga³, Keiichiro Fukazawa⁴ (1.National Institute of Information and Communications Technology, Tokyo, Japan, 2.CLEALINK TECHNOLOGY Co., Ltd., Kyoto, Japan, 3.Systems Engineering Consultants Co., Ltd., Tokyo, Japan, 4.Academic Center for Computing and Media Studies, Kyoto University, Kyoto, Japan)

Keywords:High-Speed File Transfer, Big Data, JHPCN, HpFP

With the rapid growth of data amount, big data processing imposes a substantial burden on computation, storage, and communication in data centers. The file transfer tool is used to provide the users with the ease of transferring files across the network, e.g., between data centers. In practical long distance networks, the amount of packet loss is not negligible to seriously affecting the file transfer. In general, the file transfer tool is based on transmission control protocol (TCP) and user datagram protocol (UDP). To the best of our knowledge, however, no file transfer tools based on TCP and UDP have succeeded in higher throughput than 10 Gbps on long distance networks with packet loss. In this research, we introduce a high-speed data transfer protocol for inter-data center transport network, namely high-performance and flexible protocol 2 (HpFP2). The HpFP2 is an improved version of the HpFP and is more suitable for real network environments. To accommodate different usage scenarios, we complete four operating modes of HpFP2: aggressive, fair, fast-start, and modest modes. The aggressive mode is to maximize its own throughput without regard to fairness or network stability. The fair mode is to maintain the fairness among all network connections and balance the speed of each network connection by gradually increasing the amount of data transmitted until it finds the network’s maximum carrying capacity. The fast-start mode is to improve the properties of fair mode by providing a fast and stable experience. The modest mode is to improve the estimation of the fair transmission rate and prevent the rate oscillation which is occurred by the aggressive mode. Based on HpFP2, we implement a file transfer tool, called high-performance copy (HCP), to improve file transfer performance over Japan high performance computing and networking (JHPCN). The performance of our file transfer tool is evaluated using real datasets collected from supercomputer resources. The results show that the HCP achieves notably better performance than the conventional tool for file transfer over JHPCN.