Metrics

Nothing is perfect. That can be a kind of defeatist perspective if looked at the wrong way, but it's also a source of optimism. Everything provides an opportunity for improvement if you can just stop and learn the lessons of what went wrong.

This is often easier said than done. "What went wrong" is a terribly amorphous idea, especially when dealing with something that has lots and lots of moving parts but will still lurch forward instead. Some endeavors have outright failure conditions. If a car won't start or a house falls down, then there's clearly something wrong. But more often the car is making a strange noise or the house has intermittent electrical problems. These are much harder to address than a true failure, if only because failure demands action. Persistent, systemic problems that don't actually cause failure can be tolerated, often well past the point of reason.

This is why people in expensive suits talk a lot about "metrics". In the absence of a failure scenario, you need some way to improve upon things, and the only way to consistently do that is by measuring things, changing them, then measuring again to see if there's some improvement[1]. But what should you measure? This can be surprisingly hard to figure out, and it's valuable enough to explain why those guys can afford such expensive suits.

So, take all those problems and complicate them further with ideas of "fun" and "entertainment" and you have the problem of applying this thinking to RPGs, where I would argue it's desperately needed.

First and foremost, how can you even spot a failure scenario with an RPG? Is it the game that doesn't happen? That doesn't finish? That finishes badly? Who's to say those breakdowns came from the game and note from secondary failures (perhaps in transportation or coordination). As individuals, we may have some spectacular car crashes of games we can think of that we might be able to deconstruct, but those tend to be so intensely personal and specific that there's not much room for common language. That is to say, if you want to deconstruct one of your failures, then don't expect much help.[2]

And that's the big easy part. What about games that don't fail? What concrete things can you look at and say 'this didn't go so well' so that you can try something to fix it and see if it works? What kind of metrics can you even look for in an RPG?

I don't bring this up rhetorically. I genuinely don't know, and it frustrates me intensely. There are a few vague shapes - you can track resources in a resource management game and use that as a rough guideline, but that tends to be very rough indeed[3]. You can track concrete social elements like attendance and time played, but I'm not sure what those tell us.

Arguably, the best case is probably to just pick some things, assign some numbers and fake it. If I rate "Player enthusiasm" at the end of every session from 1-5 then there's lots of room for error, but over time I am likely to get some useful trending, even if the specific numbers themselves are only so reliable.[4]

But what to measure? What should the metrics for a successful (or failed) game be? I'm intensely curious to hear people's thoughts on this.



1 - This is contrasted with just going by gut. Some people can do this, and that is truth, but many many more people THINK they can do this than actually do. And, of course, determining whether or you can is probably best not judge by your gut.

2 - 'But I can discuss it on the Internet' says the hapless optimist. And technically, that's true, and there might even be some faint insight to be gained in that fashion, if you're willing to dig and sift through the noise, but at best you'll usually get the answer for how someone else (with perfect hindsight) *would* have done it. This is kind of poisonous because, like the TSA, we see clear trails into the past and think it should have been equally obvious in the other direction, and the problem is that clearly we just missed it. From that perspective, it is too easy for the guy giving advice to be taken as a guru. And we don't need any more gurus.

3 - For example in D&D, you an track damage taken, healing surges and powers used, and use that to gauge how well you've been balancing encounters. Unfortunately, there are so many other variables (such as changes form leveling up, different monster abilities and dumb luck) that it's only so useful.

4 - There's a dirty trick implicit in this, and that is this: you get what you measure for. As such, deciding what to measure for is also an implicit declaration of what you want to see more (or less) of in a system. There's some really fascinating stuff about this and the impact of the Apgar Score (a measure of the health of newborns) in Atul Gawande's Better.