The k-factor

So far, we’ve established a way of rating a cricket team (or indeed, any other participants in a two-headed completion).  We use a rating to predict the outcome (and we’ve seen how this is done); we observe the difference between the expected and actual outcomes; and we adjust the ratings according to the result – the more unexpected the outcome, the more we move the rating by.  But in what proportion to the difference between the expected and actual results should we adjust the ratings? This is known as the k-factor.

As we’ve defined things, both expected and actual results vary between 0 and 1; and we’ve also defined a 100 point difference as implying an expected result of 2/3 for the stronger team, and 1/3 for the weaker.  So let’s suppose two teams start a series and their equally fancied, and k is set to 100.  0.5 points each is predicted when the difference is zero, so if the game is drawn, the prediction is met exactly and the ratings of both teams stay the same.  But if team A wins, it registers 1 point, and so the difference from the expected value is 0.5. With a k of 100, team A’s rating will therefore rise by 50 points, and its opponents’ rating will fall by the same.  So there’s now a difference of 100 points.  That means, on the strength of this one game alone, one has switched from predicting a drawn result in a single game, or the same number of wins each in a multi-match series, to a prediction, for example, that a three match series would end in a score of either 2-1 or 1-0.

Now, obviously, given that team A is now ahead in this particular series, it has a great chance of winning it, particularly if this series only consists of three games.  But the outcome of this particular isn’t what the new expected value is predicting.  Instead, it’s predicting the outcome of another three match series, starting from scratch, not including the game that has just been played.  And instinctively, 100 seems too large a value for k.  If two teams are supposed to be equal, it’s no real surprise that one of them might win a game between them.  This might cause one to slightly up one’s belief in that team’s ability relative to the other – but the shift in expected value feels rather too large.  With too large a value of k, ratings will be highly unstable: a team that registers a few wins in succession will see its rating soar, only for that rating to fall just as quickly whenever the team loses a match.  But a small k is also a problem.  Many wins might do little to boost a team’s rating; if a team wins a series 3-0 and this does not substantially alter expectations for the next series between the two sides, something might appear to be wrong. It’s an equivalent problem to the one we had when considering how to build a weighted average: the best estimate of a team’s ability reflects both past and more recent form. The latter is clearly the better pointer, but how do we attach relative weights?

I started with a k of 40.  Which means, after 1 match between two previously equal-rated sides that ends in a win for one of them, each side’s rating would change by 20, and if they played again, the expected value of the stronger side would increase to approximately 0.57.  A second consecutive win for the same side would increase the ratings gap from 40 to 74 and the expectation for the third match to 0.63.  There’s a law of diminishing returns here: the greater the difference we initially believe there to be between two teams, the less a win for the team we already thought was stronger is going to further increase our regard for it.  And ultimately, no matter how awesome a run of wins a team puts together, we still can’t logically expect it to get more than 1 point per game (which is why we used an exponential function to define our expectations).

Can we do better than a k of 40?  In fact, because a lot of test cricket has already been played, it’s possible to explore the use of different values of k by testing them empirically – by asking, which value of k would have produced the ratings at each point in time that cumulatively turned out to have the best predictive power of subsequent results? And so we can conclude that the best value for k, based on past evidence, is not in fact 40 (though that wasn't such a bad guess) but actually 34.  I’ll talk a bit more about this calculation later, but first of all, I need to step away from general Elo theory and go into some cricket-specific details of my implementation.

Central Limit Theorem

In the previous post, I introduced the concept of the normal distribution, as the shape of the frequency distribution of the outcome of a number of rolls of a number of die, as both numbers tend towards infinity.  This sounds like a very abstract concept, of little applicability to everyday life.  But in fact, if you look at the distribution of heights in a population, it’s quite accurately described by a normal distribution. So are results from intelligence tests.  Even the batting averages of test cricketers who’ve scored over 2000 runs (a relatively small group of men) seem approximately normally distributed, although the relationship is less perfect.  How come?  There was a theoretical basis for the distribution of die rolls.  But surely a batsman’s ability cannot be modelled in such a regular way. A batsman’s ability may depend on many factors: strength, hand-eye coordination, coaching, attitude, etc.; and there’s no a priori reason for expecting the distribution of such factors to vary normally.

The key concept here is something called the Central Limit Theorem, which states that, if one combines two independent and approximately normal distributions, the combined distribution of both variables will be more normal than the individual distributions.  Thus, one doesn’t need the individual factors that contribute to a batsman’s average to be good fits to a normal distribution, provided there are many such factors contributing to the output, for that output itself to be fairly normal.  There’s a theoretical proof of the theorem, but the ubiquity of approximately normal distributions in nature seems to be good empirical evidence in addition. Indeed, the distribution is named precisely because it’s normal to find it.

As the outcome of cricket series also depends on many independently varying factors, we can also assume that these are normally distributed; or to be precise, that the distribution of the outcomes of a large number of series each comprising a large number of games between two teams of fundamentally constant ability will be normally distributed.  If the two teams are of equal ability, the mean of the distribution – which, as the distribution is symmetrical, is also its central point and the peak of bell curve – will correspond to a result of 0.5 points per game (remember, we assign 1 point to the team that wins a match, and a half point to both teams if there’s a draw – so the mean outcome is equality, which is in fact what we mean when we stated the premise of “equal ability”). And if the difference between two teams is such that the expected outcome is in fact 2/3, we might still expect a normal distribution of results –only now the peak of the graph comes at 2/3, not at 1. 

But now we have a theoretical model for the outcomes of games.  And fitting this model allows us to answer the question we had, if a ratings gap of X corresponds to an expected outcome of 2/3 in favour of the strongest side, what does a ratings gap of 2X imply in terms of outcome? And what emerges from this is the value of the parameter B in the formula we introduced, where the expected outcome of the strongest team is equal to 1 / (B^-D + 1), where D is the ratings difference between the two teams.  As we saw, the value of B will determine the answer to the question; and the value that emerges from this analysis is our little friend e, this peculiar irrational number which, among other things, is part of the formula describing the normal distribution. A pretty decent mathematical explanation of this can be found here; note also that in chess, empirical evidence has suggested that in spite of the theory above, a normal distribution may in fact not be the best way ofmodelling actual results.  But B = e in the game of Go; and we’ll take that value for use in our system.

So we can now predict the outcome of any game between two teams with a defined rating; and compare the actual outcome to that predicted.  But how much should an aberrant result change our future expectations?  That requires us to set a new parameter, which we’ll call K.  And K will be the subject of the next post.

Normality and e

Let us consider a square, and let us define a unit of measurement such that the area of the square is 4 square units.  The length of each side of the square is thus 2 units. But supposing we define our unit of measurement differently, so that the area is 2 square units.  What is the length of the side?  We haven’t changed the square, only the unit of measurement; yet the length of the side is hard to pin down.  It’s approximately (but not exactly) 1.41, but no matter how much we improve the precision of our estimate, we can’t get the right answer.  In fact, since the time of the ancient Greeks, we’ve known that the exact answer can never be expressed as a fraction, or ratio, of two integers – you can find some fairly easy-to-follow proofs of this on Wikipedia.  Hence, we call it an irrational number.

Now, consider the family of curves defined by the formula y = B^x, where B is a constant.  Can we choose a value for B so that the gradient of the curve is exactly 1 when x = 0?  We can, but once again the answer (in this case, approximately 2.71) is irrational. Mathematicians call this number e. Now, e seems to be a pretty esoteric concept; but in fact, it crops up in many places in the real world. We’ll come back to e in a moment, but first I want to change the subject.

Consider rolling 2 die.  There are 6 possible outcomes for each dice, and thus 36 possible combinations. Six of these will result in a sum total of 7 (1 and 6, 2 and 5, 3 and 4, 4 and 3, 5 and 2 and 1 and 6).  Just one combination of the 36 (1 and 1) will result in a sum total of 2.  Random factors will determine the outcome of each roll.  But random variations tend to balance each other out over time.  Thus, we can predict from theory, that if we were to roll two die on a very large number of occasions, approximately 1/6 of all results will be 7, but only 1/36 will be 2.  And one can plot the predicted relative frequency of the different outcomes on a bar chart. I’ve included one for reference below:

Now consider that we do the same thing with four die. The most common outcome will now be 14, but we get a slightly different shape to our chart.  The new chart (see below) has more of a bell-shape:  the central portion is more highly represented with respect to the flanks. And, as there are more outcomes to depict, but we’ve plotted it on a chart of the same size, the picture is a smoother one, although we're still plotting the frequencies of discrete events (the probability that the sum of four die is 14.5 is of course zero).

Now, supposing we keep adding dice.  Gradually, as the number of die per roll increases, the chart showing the relative frequency of outcomes would look increasingly identical to a completely smooth curve, with a well-defined bell-shape.  And this curve - in effect, the distribution of outcomes we would get if we were to roll an infinite number of die on an infinite number of occasions - is known as the normal distribution. You can see exactly what one of those looks like here.  Typically, the data is “normalised”, so that the central point of the distribution has a value of 0, and a measure of the spread (called the standard deviation) is set to 1. The mathematical formula describing the shape of this distribution is complex, and I’m not going to type it out (you can find it near the top of the relevant Wikipedia page). For now I’ll notice just one thing: the formula contains that magic number e we saw earlier.

This might still seem pretty obscure: what does the distribution of an infinite number of rolls of an infinite number of die have to do with cricket?  In fact, the answer is rather more than we might expect.  And that will be our next subject.

Expectations and Exponentials

In the last post, we established that you might revise an estimate of a cricket team’s ability by comparing an actual result to that expected, if that estimate had been correct.  But what do we mean by “expected” in this context?

If a strong team plays a weak one, we expect it to win, in that a win may be the most likely result.  But even the strongest team doesn’t win every match it plays.  Perhaps we expect a strong team to win a six match series 4-1.  So if we give a point for a win and split the point in the event of a drawn game, the team would win 75% of the points, or 0.75 points per game.  Obviously a team can’t actually get 0.75 points from a single game, but thinking in this way allows us to put expected values on a continuous scale.  And if one puts one in the context of betting, the frequency of wins  becomes very important: if you judge the expected outcome correctly, and make enough bets at favourable odds, you’ll come in a winner: if odds of 4-1 are offered for a win (with the bet annulled in the event of a draw), consistently betting on a team with only a 0.25 expected value will see you ahead (with three times as many losses as wins, but four times the yield when they do win as what you lose when they do not).

And the other idea was that our ratings would be used to define our expected value.  Let’s assume equal ratings mean equal chances, and an expected value of 0.5.  And let us state that as the gap between two teams grows very large, so the expected value tends towards 1.  It can’t exceed 1; no possible team can average more than 1 point per game, as we have defined it.  But the greater the gap between two teams, the more one-sided their records.  And in fact we can define a formula that meets this criteria: in fact, we can define a whole family of formulae.  Our formula is as follows: the expected result (E) for the strongest team is equal to 1 / (B^-D+ 1), where D is the difference between the two team’s ratings, and B is any positive number greater than 1.  Any number raised to the power 0 is 1; any number greater than 1, raised to a large negative number, is little more than 0 (and becomes smaller the larger the power to which it is raised).  Thus when D is 0, the expected value is 1 / (1 + 1), i.e. 0.5; and when D is large, the expected value becomes 1 / (~0 + 1), i.e. ~1.

So now we need to set B, but firstly I’m going to add another term to the formula, A, as follows:  E(strongest team) = 1 / (B^-D/A+1).   The A term here is used to calibrate the ratings; we can consider our first formula to be identical to the second, only A has been set to 1.  But for any chosen B, there will be an A such that a fixed gap in the ratings will be equivalent to a certain expected value.  In my system, I’ve chosen an A so the E(strongest team) = 2/3 when D = 100. For the weaker team, the expected value is very easy to calculate: it’s whatever’s left that the strongest team hasn’t claimed.  So in the case of my system, a team that rates 100 points less than it’s opponent has an expected value of just 1/3.

Thus, if two teams have a rating difference of 100, one would expect a 1-0 win in a 3 test series, or a 2-1 victory, for the strongest team. This is arbitrary: by picking a different A, one could make one point difference a hugely significant one, or a 1000 point difference trivial: in this sense, it’s purely an aesthetic preference.  Nonetheless, whatever A we pick, a given D has a meaning: it may not be an average of past performance, but it corresponds to an actual predicted outcome of the chances of the two sides. Equally, in absolute terms, a rating means nothing.  The difference between two ratings is a measure of relative strength.  If one was to add (the same number of) millions to every team’s rating, the system would be unaffected. A team rated one million and one has exactly the same predicted advantage over a team rated one million as a team with a rating of one has over a team with a rating of zero.

So now we have to pick a B. And this is not arbitrary, because it determines the shape of the curve.  It’s clearly not a linear relationship: we’ve already asserted that as D tends towards infinity, the expected value tends towards 1.  But we can give a very specific example to make things clearer.   If team A has an expected value of 2/3 against team B, and team B has an expected value of 2/3  against team C, what is the expected value when team A plays team C?

Suppose B is 2.  Setting A so a difference of 100 gives an expected value of 2/3, A needs to be ~63.  And then, if D is 200 not 100, the expected value of the strongest team is ~0.9. But if B is 4, A needs to be ~115, and the expected value of the stronger team is ~0.92. Under both formulae, then, there’s a clear prediction that A is expected to get the better of C quite often, but nonetheless, the values are different.  And this is not just a calibration problem.  A 100 point difference has been defined as a real thing (namely, the difference that corresponds to a 2/3 expected value), but our system must produce a prediction for any points difference.  And if I was to pick a B so small that a 200 point difference (the sum, after all, of two 100 point differences) corresponded to only an expected value of 0.68, or so large as it corresponded to an expected value of 0.99, it would be self-evidently wrong.  That is to say, even though the model is not a linear one, the significance of the 200 point gap (or indeed, a gap of any other margin) must be a function of the significance of the defining 100 point gap, and the difference between 200 and 100. If we can’t find such a relationship, the system will work when the gap is exactly 100 points, and fail thereafter.  In fact, we’ve just defined an infinite family of such relationships; but objectively, one should be right.  And oen can think of two ways we might try to find the the right B: one is empirical, to see which B best fits the actual data, but the other approach is theoretical, and to define it from first principles.

To cut to the chase: our B is going to be approximately 2.7.  In fact, it’s going to be an irrational number, a number that can be shown to exist even though we can’t exactly write it down (a bit like the ratio of the radius of a circle to its diameter).  Because we can’t write such numbers down, mathematicians represent them by letters; and our number is known by the letter e.  But what exactly is e, and why do we choose it?  I’m not going to give you a formal proof, but to give even a cursory explanation I do need to give an introduction to the concept of the normal distribution.  And that’s the next post.

The Genius of Elo

In the previous post, we asked two questions? How do we compensate for the difficulty of a team’s schedule when we assess the team’s results?  It’s not such an easy thing to do: if you beat a supposedly strong team easily, doesn’t that just prove it wasn’t actually so strong? And secondly, how do we weight recent results against older ones? When a team records a new result, to what extent should we change our previous assessment of it?

And the Elo system provides us with a very clear model for thinking about these two points, starting with the second one, but based on how in general, we make assessments in life.  If something happens which is very unlikely according to our previous preconceptions, we adjust our preconceptions more radically than if something happens which is very close to what we expected.  If a cricket team beats a side we thought was slightly weaker than it, we’re not so very surprised, and our assessment of the two teams doesn’t change very much.  But if a supposedly weak team wallops a supposedly mighty one, we may consider this a freak result; but we are also more likely to question our prior assumptions about the two sides’ relative strength.

And the second key idea of the Elo system is this: that if we say that a team is stronger than another one, what we mean is that we expect it to do well, should the two teams play each other.  Thus we have a basic formula for adjusting a team’s ratings after a match: that the new rating of a team A,  R(A)₁, equals R(A)₀, its previous rating, plus some function of the difference between the result achieved and that expected; where the latter is itself some function of the difference between R(A)₀ and R(B)₀, the previous rating of the team’s opponents.

We’ll look into what that function might look like in the next post, but for now, I just want to explore the difference between a rating of this sort and an average.  After every game, R will change.  Even if a team was of absolutely fixed underlying ability, it would win some matches and lose some; and it’s rating would fluctuate.  We can think of the rating less as a summary of performance (a measurable thing) over a fixed time interval, but as a guess of the un-measurable quality of ability derived from the performance.  After each game, we make a new guess of the correct rating, informed by the extent to which the team’s actual performance has deviated from that we would have observed if the ratings were already perfect predictors of outcome.

Such a rating is thus very different from a statistic that tells us (as a fact) that a team has won 40% of its last 10 matches. But here’s an interesting point. How do we start operating such a system?  Before the first match we consider, how do we know what to expect? An obvious starting point is to assume all teams are equal, before we have evidence to the contrary.  But providing we are happy to give the system a little time to equilibrate, it doesn’t actually matter.  If our initially allocated ratings are poor, results will differ more from expectations; and so our originally assigned ratings will self-correct quickly. 

And thus we also solve the two problems we had previously.  A system which uses rolling, and/or weighted averages, is in danger of either over-privileging old results (so, for example, a team in decline keeps a good rating even once it is no longer performing well), or suffering from absurd instability (whereby a couple of good matches could lead to a side being top rated).  But under the Elo system, the team’s ratings will move more decisively in response to a team’s recent results when those results differ dramatically from expectations.  Thus the more a team underperforms, the greater the correction to the ratings (and the greater effective weight of more recent results).  In fact, we will still need to pick a general stability parameter; but as we will see, the system gives us a framework for at least trying to set this in a non-arbitrary way.

And our first problem is also solved.  A team with weaker opponents will have better results expected; and so will need to win more games to beat its expectations.  It is logically no more likely that playing five matches against weak opponents will improve your rating than playing five games against a strong team.  The principle of the system is thus very clean.  The mathematics will be explored next time.