Japan Geoscience Union Meeting 2018

Presentation information

[JJ] Oral

S (Solid Earth Sciences) » S-TT Technology & Techniques

[S-TT51] Creating future of solid Earth science with high performance computing (HPC)

Wed. May 23, 2018 3:30 PM - 5:00 PM 301A (3F International Conference Hall, Makuhari Messe)

convener:Takane Hori(R&D Center for Earthquake and Tsunami, Japan Agency for Marine-Earth Science and Technology), Yuji Yagi(Graduate School of Life and Environmental Sciences, University of Tsukuba, Tsukuba), Katsuhiko Shiomi(国立研究開発法人防災科学技術研究所), Chairperson:Hori Takane, Shiomi Katsuhiko(防災科学技術研究所)

3:30 PM - 3:45 PM

[STT51-01] Acceleration of crustal deformation computation using GPUs and its application to stochastic inversion analysis with geometry uncertainty

★Invited Papers

*Takuma Yamaguchi1, Kohei Fujita1, Tsuyoshi Ichimura1, Takane Hori2, Muneo Hori1, Wijerathne Lalith1 (1.The University of Tokyo, 2.JAMSTEC)

Keywords:Finite Element Analysis, OpenACC, Conjugate Gradient method, Element-by-Element method

Numerical simulations of crustal deformation have been studied with the aim of understanding the state of the crustal structures in relation to earthquake generation processes. Most studies have used analytical solutions, with the assumption that the crustal structure is a half-infinite space. However, recent studies have used the three-dimensional (3-D) finite element (FE) method at low resolutions, because simplifying the 3-D heterogeneity of crustal structures may significantly impact the results of some cases. In addition, the availability of high-resolution crustal deformation observational data and crustal structure data has increased with the improvement of observation technologies. As a result, the demand is growing for methods that can compute crustal deformation using high-resolution 3-D numerical modelling and that consider the surface geometry and heterogeneity of crustal structures. When targeting Japan, the domain of analysis is on the order of 1000 x 1000 x 100 km. If a numerical model based on high-fidelity crustal structure data with sufficient fine discretization to guarantee convergence of the numerical solution is used, the model would have more than 10,000,000 degrees-of-freedom (DOF). To handle such a massive computation cost within a realistic timeframe, there is great interest in developing fast numerical computation methods for large-scale crustal deformation computations. In these computations, most computation time is spent in solving a system of linear equations, which is the result of constitutive rules and discretization. Therefore, creating a faster solver for systems of linear equations would benefit fast numerical simulations.

Multiple crustal deformation computations enable stochastic inverse analysis, optimization, sensitivity analysis, and Monte Carlo simulation. These applications are important in considering uncertainties, including those in material properties, geometries, and inputs. However, the computation cost of such simulations increases depending on the number of repetitive computations required.

GPUs have recently become common in scientific computing. It is thought that they are broadly applicable to numerical simulations with parallel computation. Use of GPU accelerators is expected to speed up simulations. However, GPU calculations often become memory bandwidth bound computation; thus it is difficult to exhibit high performance in a straightforward implementation. Here, we propose a method for computing elastic crustal deformation using a fast solver with multiple GPUs. We modified the algorithm according to the hardware architecture in the GPU. As for the sparse matrix-vector product, which accounts for the largest proportion of the computation time, we introduced the Element-by-Element method and reduced the amount of memory access.

To test the proposed method, we estimated the coseismic slip distribution by multiple crustal deformation computations. We targeted the northeastern Japan and generated FE models which have about 80,000,000 DOF. We computed 360,000 forward analyses (360 forward analyses x 1,000 different FE models) and conducted a stochastic inverse analysis. These elastic crustal deformation computations were computed in nine days by using a GPU cluster comprising 16 CPUs (Intel Xeon E5-2695 v2) and 64 GPUs (NVIDIA K40). We calculated the average value and standard deviation of coseismic slip distribution for 1,000 cases. The standard deviation of the slip distribution was 13% of the average value. This indicates that consideration of uncertainties in geometry is significant because the obtained standard deviation is non-negligible when discussing the coseismic slip distribution and related stress change distribution. Using our proposed method, a stochastic estimation of coseismic slip distribution, with uncertainties in geometry, was computed within a realistic timeframe. In future studies, we will apply this method to the optimization of crustal structure.