Car trouble this morning kept me away from the KB, so apologies for the late post. Today, we're going back to compiling the list of GM metrics.
So rules mastery is a little bit contentious as a datapoint, but I think it's an important one because it's not really about rules mastery so much as it's about how rules knowledge and handling impacts the game. That idea is probably easier to show than explain, so the rating basically breaks down as follows.
0 - Applied rules incorrectly or spent extensive time on reference or rules arguments.
1 - Rules reference* was required but did not impede the game
2 - Rules were correctly applied without reference.
* Reference defined as having to look outside the elements in play to get information. Referencing a character sheet or GM's screen (at least a good one) is not reference, but checking a book or website would be.
Laid out that way, I'll speak right to the obvious problem - this seems to suggest that ad hoc rulings and houserules are bad, because they elicit a lower score. That is, of course, utter bullshit. Ruling on the fly is something that you usually do to make a game more awesome, and it seems odd to penalize that.
The thing to highlight is that this is not a penalty, it's a measure of friction. If the game is demanding a lot of rulings then it might still be a lot of fun (based on how awesomely you make those ruling) but it's possible that there might be some disconnect betweent he game you want to run and the game you want to be using, and this reflects that. Both of the 0 behaviors are reflective of such a disconnect.
So what about house rules? For purposes of this test, house rules _are_ rules. They're the rules your using to run the game, and so long as everyone knows what they are, they don't count as avoiding the rules. That last is a bit of a hurdle for the GM who keeps everything in her head, but if a 0 score is enough to encourage that GM to share her thinking with players, then it's good incentive.
Now, to reveal my own misgiving, I'm uncertain about the inclusion of rules arguments in this, and may ultimately remove it. While rules arguments may reveal a disconnect with the rules, I think it's more likely they're reflective of a social problem, one that is only tangentially related to the rules.
Bottom line though is that this is a measure of how much the application of rules (which is different than the actuality of the rules) impacts play. It definitely has a bias towards games with easier rules mastery (that is, light games), but since the yardstick is the impact on play, not the size of the rules, it's not as big a gap as all that. Curiously, this also biases towards good design - a well designed character sheet reduces the need for reference, and can potential bump up the score of a game as a result. It's a secondary effect, but a kind of interesting on.