4:15 PM - 4:30 PM
[MGI37-04] Development and operation of real-time earth observation data archiving and processing system on DIAS
Keywords:Earth observation data, Real-time data, Data archive, Big data, User interface
Data Integration and Analysis System (DIAS) was launched in 2006 as part of the Earth Observation and Ocean Exploration System, which is one of five National Key Technologies defined by the 3rd Basic Program for Science and Technology of Japan. DIAS is operated as a system to coordinate the cutting-edge information science and technology and the various research fields addressing the earth environment, to construct data infrastructure that can integrate earth observation data, numerical model outputs, and socio-economic data effectively, to create knowledge enabling us to solve the earth environment problems, and to generate socio-economic benefits.
On DIAS, we are developing and operating real-time or quasi-real-time data archiving processing system using variety of characteristic technical approach, which is different from usual earth observation data archiving system because of the characteristic feature of data.
First, as a feature of real-time earth observation data, it is quite difficult to re-acquire those data in case we fail to acquire at the appropriate timing.For example, GPV(Grid Point Value) data which is generated by Meteorological Agency in Japan and provided by Japan Meteorological Business Support Center is stored only the latest one week data because of the amount of data volume. After a period of time the data is holded, the acquisition of those data become no longer very difficult to acquire.Further, in the case of a system which can provide real-time live camera images, we cannot acquire any historical data after the next data is obtained because the image data archived at provider is overwritten every acquisition time.As the reason of failure factors of such acquisition, trouble of acquisition system program, stopping associated with routine maintenance, trouble of the system of data provider, network factor and etc. are considered, sufficient countermeasures are required to acquire data certainly within a predetermined period of time.
Secondly, because many of real-time data are periodically observed or generated, and continuously provided new data, our system must terminate all kind of processes within the delivery interval.This is not considered so much about the system of archiving general earth observation data, it is necessary to develop a processing system which is considered deeply about the limitation of delivery interval, when we start to archive and start a service for providing a data acquired regularly.In addition, it is assumed that it is possible to get late acquisition fails for some reason as described above, consider the mechanism to recover the delay is also required.
The third point is to manage the information of consistency of data, missing of data.In the archive of earth observation data, data is considered to be lacking in the various factors of creating side, providing side, middle path of transfer, and acquisition side, which is not limited in the case of real-time data.
In a fourth aspect, when performing the cooperation with an application that processes real-time data in real time, there is a point that must be considered, including the treatment protocol for data to be delivered, especially delayed.Generally, because applications with real-time data in real-time require the immediate results in most cases, including the variations of acquisition time of each area and synchronization of the data and application describes above, development of processing policy is very important.
In this paper, we introduce our real-time earth observation data archiving and processing system on DIAS with some specific examples, considering these points describes above.
On DIAS, we are developing and operating real-time or quasi-real-time data archiving processing system using variety of characteristic technical approach, which is different from usual earth observation data archiving system because of the characteristic feature of data.
First, as a feature of real-time earth observation data, it is quite difficult to re-acquire those data in case we fail to acquire at the appropriate timing.For example, GPV(Grid Point Value) data which is generated by Meteorological Agency in Japan and provided by Japan Meteorological Business Support Center is stored only the latest one week data because of the amount of data volume. After a period of time the data is holded, the acquisition of those data become no longer very difficult to acquire.Further, in the case of a system which can provide real-time live camera images, we cannot acquire any historical data after the next data is obtained because the image data archived at provider is overwritten every acquisition time.As the reason of failure factors of such acquisition, trouble of acquisition system program, stopping associated with routine maintenance, trouble of the system of data provider, network factor and etc. are considered, sufficient countermeasures are required to acquire data certainly within a predetermined period of time.
Secondly, because many of real-time data are periodically observed or generated, and continuously provided new data, our system must terminate all kind of processes within the delivery interval.This is not considered so much about the system of archiving general earth observation data, it is necessary to develop a processing system which is considered deeply about the limitation of delivery interval, when we start to archive and start a service for providing a data acquired regularly.In addition, it is assumed that it is possible to get late acquisition fails for some reason as described above, consider the mechanism to recover the delay is also required.
The third point is to manage the information of consistency of data, missing of data.In the archive of earth observation data, data is considered to be lacking in the various factors of creating side, providing side, middle path of transfer, and acquisition side, which is not limited in the case of real-time data.
In a fourth aspect, when performing the cooperation with an application that processes real-time data in real time, there is a point that must be considered, including the treatment protocol for data to be delivered, especially delayed.Generally, because applications with real-time data in real-time require the immediate results in most cases, including the variations of acquisition time of each area and synchronization of the data and application describes above, development of processing policy is very important.
In this paper, we introduce our real-time earth observation data archiving and processing system on DIAS with some specific examples, considering these points describes above.