[1P1] [General Session] 3. Data Mining

[1P1-01] A Sampling Method based on Generalized Relative Square Error to Emphasize Low Probability Events

Hiroshi Hasegawa¹, 〇Tomomi Nakamura¹, Takashi Washio² (1. Dept. Math and Sciences, Ibaraki University, 2. ISIR, University of Osaka)

Keywords:data sampling , generalized relative square error, events of low probability , Large deviation theory, Wang--Landau algorithm

A method of data sampling from a huge data set is discussed. We introduce a generalized relative square error to emphasize low probability events and figure out the best sampling weight to reduce the error. Our arguments are based on the large deviation theory. Large reduction in the generalized relative square error was numerically confirmed for the best sampling weight. We also propose to use Wang-Landau algorithm in data sampling. This algorithm is not only efficient to estimate a distribution of the original data, but also useful in data sampling to suppress the statistical errors.