So, what then is the value of forecasting, if it can never be proven right or wrong? That is, if we're saying there's an 80% chance of Republican control of the Senate on election day, but the Democrats wind up winning, the forecaster can just say, "This was part of the 20%." So what was the value of the exercise?
Over time, when he says something has an 80% chance, it should happen around 80% of the time and not happen 20% of the time. With a large enough sample size, we can test to see if his model is good. If the things he has have an 80% probability only happen 60% of the time, his model sucks. Similarly, if they happen 90% of the time, his model sucks.
Remember, he's supposed to be "wrong" 20% of the time in this scenario. If he's only "wrong" 5% of the time, his model is awful, even if he probably wouldn't get the discredit.
