ESRI Discussion Paper Series No.302
Evaluating Density Forecasts with Applications to ESPF

Kanemi Ban
Professor, Graduate School of Economics, Osaka University
Masaaki Kawagoe
Special Fellow, Economic and Social Research Institute, Cabinet Office
Hideaki Matsuoka
Senior Economist, Japan Center for Economic Research


This paper evaluates density forecasts using micro data from the ESP forecast (ESPF), a monthly survey of Japanese professional forecasters. The ESPF has collected individual density forecasts since June 2008. We employ two approaches, Probability Integral Transform (PIT) and Ranked Probability Score (RPS). First, we apply Berkowitz’s (2001) test to individual density forecasts produced every June. We fail to reject the independency in FY 2010 and 2011 real GDP growth rates. As for CPI inflation rates, we reject the independency in all the samples during FY 2008 to 2011, but fail to reject it if the sample is limited to a half with better forecast performance. The result may ensure individual densities coincide with unobserved true data generation process of the actual outcomes. Second, we calculate RPS, following Kenny, Kostka, and Masera (2012), and compare the Mean Probability Distribution (MPD), the average of individual densities, with three benchmarks -- Uniform, Normal and Naïve distributions -- and individual density forecasts. The MPD turns out to be a “good” density: it beats the benchmarks in most cases and ranks about fifth out of around 35 participants every year. Subjective judgments added to the MPD are likely to deteriorate the performance in the case of CPI inflation rate, but to improve in the case of real GDP growth rate.

Structure of the whole text(PDF-Format 1 File)

  1. Full Text別ウィンドウで開きます。(PDF-Format 477 KB)
  2. page1
  3. page2
    1. Introduction
    1. page2
      Table 1 Record of Best 5
  4. page4
    2. Methodology
    1. page4
      2.1 PIT (Probability Integral Transform)
    2. page6
      2.2 Scoring the Densities
  5. page8
    3. Data
    1. page9
      Figure 1 MPDs and Realised Values
    2. page10
      Figure 2 Fan Charts
    3. page12
      Figure 3 Number of Bins Used
    4. page12
      Figure 4 Hitting Ratio of Density Forecast
  6. page13
    4. Results
    1. page13
      4.1 Independence Test
      1. page13
        Table 2 Number of Failures by Period
      1. page14
        Table 3 Estimation Results
    2. page14
      4.2 Results of RPS calculations
      1. page16
        Figure 5 Benchmark Density Forecasts
      1. page17
        Table 4 Results of RPS Calculations
  7. page18
    5. Conclusion
  8. page19
  9. page20
  • 1-6-1 Nagata-cho, Chiyoda-ku, Tokyo 100-8914, Japan.
    Tel: +81-3-5253-2111