Japan Geoscience Union Meeting 2023

Presentation information

[J] Oral

S (Solid Earth Sciences ) » S-TT Technology & Techniques

[S-TT43] Creating future of solid Earth science with high performance computing (HPC)

Mon. May 22, 2023 9:00 AM - 10:15 AM International Conference Room (IC) (International Conference Hall, Makuhari Messe)

convener:Takane Hori(Japan Agency for Marine-Earth Science and Technology), Yuji Yagi(Graduate School of Life and Environmental Sciences, University of Tsukuba, Tsukuba), Katsuhiko Shiomi(National Research Institute for Earth Science and Disaster Resilience), Takanori Matsuzawa(National Research Institute for Earth Science and Disaster Resilience), Chairperson:Takane Hori(Japan Agency for Marine-Earth Science and Technology), Katsuhiko Shiomi(National Research Institute for Earth Science and Disaster Resilience)

10:00 AM - 10:15 AM

[STT43-05] TomoATT : A new HPC-ready open-source project of Eikonal equation solver based adjoint state traveltime tomography for a large-scale imaging of subsurface velocity heterogeneity and seismic anisotropy.

*Masaru Nagaso1,2, Jing Chen1,2, Ping Tong1,2, Shucheng Wu1,2 (1.Nanyan Technological University/School of Physical and Mathematical Sciences, 2.Earth Observatory of Singapore)

Keywords:HPC, Adjoint-state traveltime tomography, Inversion analysis, Fast sweeping method

We have started an open-source project “TomoATT”, which implements Adjoint-state Traveltime Tomography (ATT) for revealing velocity heterogeneity and seismic anisotropy. The main objective of this project is to apply ATT to large-scale problems which require HPC systems. For this purpose, we use an eikonal equation solver for forward/adjoint simulation which requires much less computational resources than wave equation based solvers. By this solver, the anisotropic eikonal equation is solved in spherical coordinates using a high-order fast weeping method. Then the Fréchet derivatives of the objective function are calculated based on the computed traveltime/adjoint fields. Optimization process is further implemented by a step-size controlled gradient descent method with multi-grid model parameterization technique.

For the implementation, we introduced hybrid multilayer parallelization, i.e. the Fréchet derivative for multiple seismic events are calculated simultaneously. In the calculation of each event, the global domain may be divided into subdomains as an MPI inter-node parallelization which removes the memory size limit of a computer. Then finally in each (sub) domain, the node values on a sweeping surface are calculated parallelly with MPI Shared Memory, which eliminates the communication cost between MPI processes. In addition to this MPI parallelization scheme, we also introduced a memory relocation and Single Instruction, Multiple Data (SIMD) parallelization for both AVX and ARM SVE in order to mitigate an inefficient memory access pattern of stencil based computation.

We performed benchmark tests on a local machine and HPCs including Fugaku. The result shows a good scaling for small to large computation grids. Finally, we apply this numerical tool to real seismic data of the California + Nevada region with massive amounts of seismic arrival time data.