Japan Geoscience Union Meeting 2014

Presentation information

Print

Oral

Symbol M (Multidisciplinary and Interdisciplinary) » M-GI General Geosciences, Information Geosciences & Simulations

[M-GI37_29AM1] Earth and planetary informatics with huge data management

Tue. Apr 29, 2014 9:00 AM - 10:30 AM 413 (4F)

Convener:*Eizi TOYODA(Numerical Prediction Division, Japan Meteorological Agency), Yasuhiro Murayama(National Institute of Information and Communications Technology), Junya Terazono(The University of Aizu), Tomoaki Hori(Nagoya University Solar Terrestrial Environment Laboratory Geospace Research Center), Kazuo Ohtake(Japan Meteorological Agency), Mayumi Wakabayashi(Kiso-Jiban Consultants Co.,Ltd), Takeshi Horinouchi(Faculty of Environmental Earth Science, Hokkaido University), Susumu Nonogaki(Geological Survey of Japan, National Institute of Advanced Industrial Science and Technology), Chair:Kazuo Ohtake(Japan Meteorological Agency)

9:15 AM - 9:30 AM

[MGI37-02] High-speed File Transfer Tool with the Gfarm File System

*Hidenobu WATANABE1, Takashi KUROSAWA2, Ken T. MURATA1 (1.National Institute of Information and Communications Technology, 2.Hitachi Solutions East Japan, Ltd.)

Keywords:広域分散型ストレージ, Gfarm, 高速ファイル転送, UDT

A distributed storage system of scale-out type is gradually being used in the High Performance Computing (HPC) to store large scale data. NICT is also running an about 3 petabyte-scale (PB) distributed storage system with the Gfarm file system and a 10Gbps Layer 2 network (JGN-X) in Japan. Gfarm is open source software of a distributed file system for a petabyte-scale grid computing, and has been adopted as a shared storage of the High Performance Computing Infrastructure (HPCI). When Gfarm copies data between storage servers in long-distance network, it uses a multiple TCP streaming technique to transfer data faster because TCP single streaming is known to produce a low network throughput in a long distance network. However, efficiency of high-speed by the technique becomes low as more distant.We developed a high-speed file transfer tool worked with Gfarm. The tool adopts the UDT protocol as a data transfer protocol and has a control function for a parallel data transfer. UDT is a reliable UDP based application level data transport protocol over wide area high-speed networks, and uses UDP protocol to transfer bulk data with its own reliability control and congestion control mechanisms. In fact, UDT can provide a high network throughput than TCP in a long distance network. We explain our tool and report the performance results of the tool in basic evaluation.