Japan Geoscience Union Meeting 2014

Presentation information

Oral

Symbol M (Multidisciplinary and Interdisciplinary) » M-GI General Geosciences, Information Geosciences & Simulations

[M-GI37_29AM1] Earth and planetary informatics with huge data management

Tue. Apr 29, 2014 9:00 AM - 10:30 AM 413 (4F)

Convener:*Eizi TOYODA(Numerical Prediction Division, Japan Meteorological Agency), Yasuhiro Murayama(National Institute of Information and Communications Technology), Junya Terazono(The University of Aizu), Tomoaki Hori(Nagoya University Solar Terrestrial Environment Laboratory Geospace Research Center), Kazuo Ohtake(Japan Meteorological Agency), Mayumi Wakabayashi(Kiso-Jiban Consultants Co.,Ltd), Takeshi Horinouchi(Faculty of Environmental Earth Science, Hokkaido University), Susumu Nonogaki(Geological Survey of Japan, National Institute of Advanced Industrial Science and Technology), Chair:Kazuo Ohtake(Japan Meteorological Agency)

10:15 AM - 10:30 AM

[MGI37-06] Comparison of grid data formats in meteorology: the reason for indexed sequential access method (ISAM) used in JMA

*Eizi TOYODA1 (1.NPD, Japan Meteorological Agency)

Keywords:GRIB, netCDF, NuSDaS, grid data, indexed sequential access method

Many format standards are used for grid data of simulation such as numerical weather prediction (NWP). Wright and Gao (2008) argued there are direct-access and sequential formats, and a choice is needed between fastness of partial read and compactness of data files. However, a JMA's local standard NuSDaS (Toyoda, 2001) uses the third category, ISAM (indexed sequential access method) which achieves both fastness and compactness. These three types of file formats are comared (Table 1).In the operational NWP (1) data structure is often sparse, (2) each file is written by a single process, and (3) many subsequent processes read a part of data tile. In this situation the weakness (cost of indexing) is not outstanding and the strength (fastness and compactness) are enjoyed. It is also noted that the weakness of sequential access (full scan for partial data) will be aggravated in the future computing with larger data size.ReferencesBruce WRIGHT and Feng GAO, 2008: GRIB vs NetCDF: Evaluation of the Technical Aspects. WMO ET-ADRS Doc.2.3(1) http://goo.gl/AFrslsTOYODA Eizi, 2001: NuSDaS: Numerical Prediction Standard Data-set System. JpGU presentation A2-011 http://goo.gl/JE0a3M