Some Reversion to the Mean – Review of Predictions from Gameweek 9

I have expressed some concern in previous weeks that Fantasy Football Robot is not predicting as well as it should do, as this ugly table makes clear:

GW	% of Predictions Correct
5	15.8%
6	13.9%
7	15.2%
8	11.5%
9	?

I wondered whether I had built or validated the models incorrectly (I wonder that permanently, to be honest, see below*). Then gameweek 9 came along and brought with it a healthy dose of reversion to the mean to partly allay my fears:

Goalies, 6 of 9 predictions correct, 66.7% precision.

Defenders, 11 of 20 correct, 55% precision.

Midfielders, 0 of 3 correct, 0% precision.

Forwards, 1 of 2 correct, 50% precision.

Overall, 18 out of 34 correct. 52.9% precision.

Only the Midfielders model failed. The other three performed admirably.

I say ‘partly allayed my fears’ because overall precision this season is now 30% for Goalkeepers, 23% for Defenders, 7% for Midfielders (!) and 36% for Forwards. These compare unfavourably with the simulated (out of sample) precisions: 34% for Goalies, 55% for Defenders, 29% for Midfielders and 43% for Forwards. The model is lagging in each position, strikingly so for Defenders and Midfielders. However, this week reminded me of the importance in judging experiments over their full term. In his book Bad Pharma, Ben Goldacre describes the deficit to science when experiments are allowed to be concluded early or extended arbitrarily (albeit in a much more important field than fantasy football). Time horizon is important and the time horizon for this experiment is the 2013/14 season!

That is not to say that I will stop trying to improve the models. I am monitoring some changes to the Goalies model and I hope to get cracking on Defenders soon.

Points-wise, the squad of the 15 highest probability players, FFR United, scored 38 points. Random Wanderers scored 36 points and the team of the most expensive players, Moneybags City scored 78 points (but cost £131.5m).

* My previous boss was an exacting man and he used to say “Only the paranoid survive”. I used to consider the parallel parable “Only the paranoid die of stress-induced coronary failure”. These days, I like to think I fall in between two stalls: worry about stuff, but always maintain perspective.

This week’s summaries are at: http://fantasy-football-robot.appspot.com/appraise?gw=9 and http://fantasy-football-robot.appspot.com/teams?gw=9