10:15 AM - 10:30 AM
[SSS27-02] Proper scoring systems available for probability forecasts targeting rare phenomena
Keywords:benchmark forecast, earthquake prediction, extended Brier's score, information gain, probability forecst, proper scoring rule
Necessary conditions for ensuring that newly introduced method can improve forecast are, existing proper scoring system and that the new method marks better score than the present method do. Murphy and Epstein (1967) pointed out that "all proper scoring system should encourage the meteorologists to make his probabilities correspond with his true belief." Probability Score (Brier, 1950; hereinafter called BS), which is well used for evaluating such as probability of precipitation forecasts, satisfies the mathematical term of the proper scoring rule; but, information gain (Kullback and Leibler, 1951) does not. The effort to score high in an improper scoring rule does not mean honest activity aiming for a better forecast in general, because the rule do not ensure forecasters to get the highest score in case they make forecast with their true belief. Therefore, we should become very careful about interpretation of the measurement or comparison of the forecast based on various information criteria and other improper scoring rule, although they still are very widely performed especially in the field of earthquake prediction study.
BS is defined as the expectation of the mean square error between forecasted probability and the existence or non-existence of intended phenomenon. The existence and non-existence are equally weighted. On the other hand, the users tend to give importance to the forecasts of relatively high probability and the cases of the existence, if the intended phenomena of the forecast is rare. Proper scoring systems available for probability forecasts targeting rare phenomena like large earthquakes are discussed below.
Proper scoring rule have to satisfy Es(x,c)=pS1(x,c)+(1-p)S0(x,c) (eq.1), and Es(p,c)>Es(x,c) for all x≠p (eq.2), where p, f, i, c, S_, and Es are probability based on true faith by a forecaster, forecasted value, existence (0) or non-existence (1), benchmark forecast such as a probability based on basic statistics, score, and expected score, respectively. The rule also have to satisfy fair condition from the viewpoints to give higher score for more difficult issue to forecast, i.e. S0(c,f)≡S1(1-c,1-f) (eq.3), and ∂S1/∂f|c=const>=0, ∂S1/∂c|f=constThe solutions of eq.1 and eq.3 satisfies S1(f,c)=-(i-f)B'(f,c)+(i-x)B'(x,c)-B(f,c)+B(x,c), A=d2B/df2 (eq.5), where A is any function.
A proper scoring system with suitable characteristics can be obtained after solving eq.5 in a certain boundary conditions and A, and if S_ satisfies eq.2 and 4.
For the first example, from the conditions about scores of benchmark forecast S_(c,c)≡0, score of perfect forecast S_(i,c)≡1, A=-2 and S_(f)=1-(i-f)2 (eq.6) are derived. 1-S_(f) is BS.
For the other example, by using the boundary condition for expected score of the perfect forecast (f=i) Es,p≡1 instead, and assume that A is the expression of degree 0 of f, A=-2/c(1-c) and S_(f,c)={(i-c)^2-(i-f)^2}/c(1-c) (eq.7) are derived. The fact (1-S_(f))/4 is equivalent to BS in case of c=1/2 shows that eq.7 can be interpreted as an extended form of Brier's Score.
General solution for proper scoring systems, and particular solutions, so-called extended Brier's Score, available for probability forecasts targeting rare phenomena like large earthquakes are mathematically derived as I discuss above.
In the meeting, the author would like to introduce the details of derivation of equations and to discuss on the problem for the practical application of the scoring systems. Besides, mathematical solutions of proper scoring rules for warnings, which is a kind of binary forecast, has already been discussed based on expected-utility theory (Hayashi, 2014, JpGU).
BS is defined as the expectation of the mean square error between forecasted probability and the existence or non-existence of intended phenomenon. The existence and non-existence are equally weighted. On the other hand, the users tend to give importance to the forecasts of relatively high probability and the cases of the existence, if the intended phenomena of the forecast is rare. Proper scoring systems available for probability forecasts targeting rare phenomena like large earthquakes are discussed below.
Proper scoring rule have to satisfy Es(x,c)=pS1(x,c)+(1-p)S0(x,c) (eq.1), and Es(p,c)>Es(x,c) for all x≠p (eq.2), where p, f, i, c, S_, and Es are probability based on true faith by a forecaster, forecasted value, existence (0) or non-existence (1), benchmark forecast such as a probability based on basic statistics, score, and expected score, respectively. The rule also have to satisfy fair condition from the viewpoints to give higher score for more difficult issue to forecast, i.e. S0(c,f)≡S1(1-c,1-f) (eq.3), and ∂S1/∂f|c=const>=0, ∂S1/∂c|f=constThe solutions of eq.1 and eq.3 satisfies S1(f,c)=-(i-f)B'(f,c)+(i-x)B'(x,c)-B(f,c)+B(x,c), A=d2B/df2 (eq.5), where A is any function.
A proper scoring system with suitable characteristics can be obtained after solving eq.5 in a certain boundary conditions and A, and if S_ satisfies eq.2 and 4.
For the first example, from the conditions about scores of benchmark forecast S_(c,c)≡0, score of perfect forecast S_(i,c)≡1, A=-2 and S_(f)=1-(i-f)2 (eq.6) are derived. 1-S_(f) is BS.
For the other example, by using the boundary condition for expected score of the perfect forecast (f=i) Es,p≡1 instead, and assume that A is the expression of degree 0 of f, A=-2/c(1-c) and S_(f,c)={(i-c)^2-(i-f)^2}/c(1-c) (eq.7) are derived. The fact (1-S_(f))/4 is equivalent to BS in case of c=1/2 shows that eq.7 can be interpreted as an extended form of Brier's Score.
General solution for proper scoring systems, and particular solutions, so-called extended Brier's Score, available for probability forecasts targeting rare phenomena like large earthquakes are mathematically derived as I discuss above.
In the meeting, the author would like to introduce the details of derivation of equations and to discuss on the problem for the practical application of the scoring systems. Besides, mathematical solutions of proper scoring rules for warnings, which is a kind of binary forecast, has already been discussed based on expected-utility theory (Hayashi, 2014, JpGU).