[M-GI37_29AM1] Earth and planetary informatics with huge data management

[MGI37-06] Comparison of grid data formats in meteorology: the reason for indexed sequential access method (ISAM) used in JMA

*Eizi TOYODA¹ (1.NPD, Japan Meteorological Agency)

Keywords:GRIB, netCDF, NuSDaS, grid data, indexed sequential access method

Many format standards are used for grid data of simulation such as numerical weather prediction (NWP). Wright and Gao (2008) argued there are direct-access and sequential formats, and a choice is needed between fastness of partial read and compactness of data files. However, a JMA's local standard NuSDaS (Toyoda, 2001) uses the third category, ISAM (indexed sequential access method) which achieves both fastness and compactness. These three types of file formats are comared (Table 1).In the operational NWP (1) data structure is often sparse, (2) each file is written by a single process, and (3) many subsequent processes read a part of data tile. In this situation the weakness (cost of indexing) is not outstanding and the strength (fastness and compactness) are enjoyed. It is also noted that the weakness of sequential access (full scan for partial data) will be aggravated in the future computing with larger data size.ReferencesBruce WRIGHT and Feng GAO, 2008: GRIB vs NetCDF: Evaluation of the Technical Aspects. WMO ET-ADRS Doc.2.3(1) http://goo.gl/AFrslsTOYODA Eizi, 2001: NuSDaS: Numerical Prediction Standard Data-set System. JpGU presentation A2-011 http://goo.gl/JE0a3M