Measuring skill/luck ratio in board games

[Jimmy V.]

Hi all,

I have just published a short study about measuring skill versus luck in games on Reddit.

The study uses stats coming from Happy Meeple platform so you will certainly find it interesting.

An up vote on the Reddit thread is appreciated.

Feel free to react there or here!

[Gues(s) T.]

Very interesting topic and very interestic article. Worth a second, slower reading (plus commentaries). Thanks a lot for posting this hint.

[Mike M.]

Indeed very interesting.  I have thought on this topic quite a bit and have my own opinions (I call them this because I have very little evidence to support them) which are much too lengthy for me to post today. 

Great read!

[Mike A.]

Cool post, though I must admit I don't understand much of it. 😛
Here's a question--at one point deep into the thread you said you can think of two reasons why a coefficient (or some term) is used to allow quick movement in ratings. What are those? I understand why that would be the case for games like chess where no luck is involved. An inferior player should rocket upwards when they beat a superior. For games like Lost Cities, however, it seems like the gain should be much, much smaller.

[Jimmy V.]

Hi Mike A.,

I said on the Reddit original post that I could see at least two good reasons for having fast evolving ratings (big K coefficient rather than small). And later I explained more or less my thoughts.

Basically, noone wants new players to be discouraged. So their rating needs to evolve quickly if they are somewhat good. Actually, we use an even bigger K for new players to make this even truer. The other reason why I think a (too) big K is better than a (too) small K is that it brings more variety, more fun to the game. Even if rating variations are sometimes a bit too harsh or a bit too positive, at least they give life to your experience on the platform. Highs and lows are good for fun I think, as long as it is not going too far.

The ideal coefficient K is one that optimizes the prediction value of the rating. In other words, the best K is the one that leads to a rating that can predict future outcomes the best. If K is too small or too big, players ratings are less accurate, either because they don't reflect players progress fast enough or because they take latest results too much into consideration. In Chess for instance, I saw a document explaining that the ideal K was 24 when the one they used was 15 or 20.

I hope this answers your questions. Rating systems are a fascinating world.

PS: the FIFA (football) ratings are absolutely awful. Instead of using a ELO system or something similar they devised a completely stupid independent system that has so many flaws it is hard to believe.

[Faabijan P.]

Great read!

[Geraldine M.]

Not sure I agree with this thinking about advancing new players much more quickly making everything more fun.

For example, the current leader on the Lost Cities board is a very new player, who has plaed many fewer matches than other people in the Top 10.

1. I feel that this devalues the efforts of the others, who have had to work much harder to get - and remain - where they are.

2. I wonder where the fun/incentive is for that player now? The only way for them to go in that game now is down.

(Full disclosure: I am one of those other Top 10 players, but I am really not saying this in a 'Promote me!' way.When I was new, I would not have wanted to soar too quickly, and am happy to inch my way up now.)

[Jeff M.]

There is the "inactivity" mechanism on the rankings.  If you don't play for a while, you lose xx many points.  I have recently gone back and played every game, but if memory serves me right the one I hadn't played in a long while had me at -100 points which dropped me significantly in the ranks.

You should be able to see this if you look to the left where the ranking calculations are done.   Pick a game you haven't played in a long time.

In other words - if you don't play your ranking WILL drop.

[Geraldine M.]

@Jeff: Perhaps you misunderstood me, or perhaps we have very different ideas of 'fun'!

To me, HAVING to keep playing in order to combat the inactivity penalty of 150 points is not the same as WANTING to play more, e.g. in order to improve one's position to a higher rank than previously achieved.

If success comes too quickly, and then a new player finds themselves slipping back down the rankings, because their initial position did not reflect their performance, but was just the product of an 'encouraging' algorithm, I don't think that will be motivating at all.

But I guess different kinds of motivation appeal to different players, so I can't speak for everyone.

[Daiana S.]

Not sure I agree with this thinking about advancing new players much more quickly making everything more fun.

For example, the current leader on the Lost Cities board is a very new player, who has plaed many fewer matches than other people in the Top 10.

1. I feel that this devalues the efforts of the others, who have had to work much harder to get - and remain - where they are.

2. I wonder where the fun/incentive is for that player now? The only way for them to go in that game now is down.

(Full disclosure: I am one of those other Top 10 players, but I am really not saying this in a 'Promote me!' way.When I was new, I would not have wanted to soar too quickly, and am happy to inch my way up now.)

I agree. I just saw a white meeple 1400 elo getting more than 40 (yes 40) points after winning ONE single match against a 1437 elo, which is even more than the difference between elos. In the meantime, when trying to get a black one meeple, every match with this small difference in elos will score less than 10 points for me.

[Jimmy V.]

If success comes too quickly, and then a new player finds themselves slipping back down the rankings, because their initial position did not reflect their performance, but was just the product of an 'encouraging' algorithm, I don't think that will be motivating at all.

Definitely, but we don't want that either. The early ELO swing boosts fade rapidly and steadily. And after 24 games, there is no boost any more. The initial boost is 2.5x the normal variation. After a few games it is less than 2.

I agree. I just saw a white meeple 1400 elo getting more than 40 (yes 40) points after winning ONE single match against a 1437 elo, which is even more than the difference between elos. In the meantime, when trying to get a black one meeple, every match with this small difference in elos will score less than 10 points for me.

Please bear in mind that initial ratings are provisional. Noone will show in the rankings list before having played 5 games. And when a new players wins or loses lots of points, it does not affect their opponent neither positively nor adversely. So it does not really matter that we show a win of 40 points. We could also hide this variation and start displaying the player's rating after 5 games.

If someone wins his first 5 games, we can safely assume that their real rating is closer to 1600 than to 1450. That's what the system does.

Also, the boost goes both ways. If the new player is really bad, why would we force them to play "strong" bots all the time when they should be playing weaker bots. It is necessary to seed their ranking as quickly as possible to their true value. A good player will have the same issue. If you are a very good player and wins 20 times in a row against bots that are too weak for you, that is not good for anyone.

As I said, the main idea of the initial boost is to seed the player as quickly as possible so that they can be matched with the best bot-opponent.


PS: I tried to use the "they" pronoun. Hopefully, I have done it the correct way.

[Jamie A.]

PS: I tried to use the "they" pronoun. Hopefully, I have done it the correct way.

Lol. I think so - it read just like Shakespeare!   

[Jeff M.]

Well done!  and I only offer this extra hint since you surprised me by admitting that English wasn't your first language -

Almost always, "his" can be replaced with "their" to be even more neutral.

[Jimmy V.]

Thanks Jeff and Jamie!

Lol. I think so - it read just like Shakespeare!

I hope it did not sound like Shakespeare! I wanted to be understandable.


NB: this might be a bad joke. In any case, for a foreigner like me, Shakespeare is just impossible to read.

[Jamie A.]

Thanks Jeff and Jamie!

Lol. I think so - it read just like Shakespeare!

I hope it did not sound like Shakespeare! I wanted to be understandable.

NB: this might be a bad joke. In any case, for a foreigner like me, Shakespeare is just impossible to read.

Hahahaha,  Nope, it was a good joke!  >>> http://www.happymeeple.com/forum/happy-meeple-community/feature-request/msg940/#msg940