Thursday, June 9, 2011

A Tale of Two Metrics

I was originally thinking of knocking our to metrics (Energy and Responsiveness) today before I realized one of them didn't work. Let me lay them out and maybe you can see why that happened.

Energy, which in my head I was kind of calling "Jazz" was a measure of how enthusiastic and engaged the table was at the end of the games.

0 - play has been flat or bad-tired.
1 - Play went ok
2 - Everyone's totally jazzed!

Bad-tired, btw, is important to distinguish from good-tired, which can fall under Jazzed. Bad tired is just beat and unresponsive. Good tired is the end-of-a-marathon kind of tired, where you're wiped but ecstatic.

Responsiveness is an idea that, like most of these, distilled from a number of other points and which might also be called flexibility. How well did the GM respond to player actions and incorporate player feedback and response?

0 - Everything went exactly as planned. Player diversions were brought back quickly into line.
1 - Player's surprised the GM, but GM rolled with it.
2 - Unexpected Player decisions dramatically impacted play in a non-punitive fashion.

Note the emphasis on surprise and unexpected in that. If the GM offers the players a choice and he's ready for the choice they make, that's things going as planned - that is to say, 0 does not automatically equate to a railroad. The non-punitive qualifier on 2 is probably unnecessary, but is just there for the GM who's "responsiveness" takes the form of punishing player choices (which is a total 0 move).

Ok, so given these two, energy and responsiveness, which one did I discard? Obviously, energy is something incredibly critical to judging how well a game went, while it's entirely possible to have an awesome game with a low responsiveness score, especially if the GM prepares well. So given that, why is it energy I'm dropping on the floor?

The answer to this is something which, I think, casts a light on why a lot of the metrics may seem less important than the things which determine how well a game went. Specifically, it's actionable. Consider: if your game has a low energy and the cause is not something obvious and external (like everyone being tired or hungover) then what steps do you do change that, to move a game from blah to jazzed? There's no one answer to that, in part because energy is an _outcome_ not something the GM _does_. Energy maybe a good thing to check to ask yourself if a game went well, but it's not useful to check if you're trying to figure out what you did.

In contrast, if I've got a low responsiveness and I want to change that, it's very easy to suggest a course of actions, even if it's as simple as "Listen to your players, respect their choices, and be prepared for them to take things in unexpected directions". Yes, those points can all be drilled into further - that's actually part of the point - but they're specific points with a specific goal. As such, they're actionable, which important to the ultimate goal of this, which is to say to be able talk in terms of things a GM can actually do rather than in terms of things they want to have happen.

The downside of that approach is that we end up with intuitive disconnects like this. Energy feels more important - it _feels_ like something we should be measure, in part because it reflects the outcome of many other successes and failures, so it seems like it should be a rich datapoint. The problem is (and this gets even nerdier) that it's actually a very lossy signal. Let's take three ways a game might be awesome - The GM might be brilliantly engaging, the adventure might be incredibly well designed, or the group might just really click with one another. Any one of those things, or a combination of them, could result in everyone being jazzed at the end of a game. But the fact that people are jazzed does not tell me which of those things happened. I might be able to intuit the answer form my recollection of the game, but even if I'm right, knowing that everyone was jazzed doesn't help me replicate it.

So energy is important, and in fact I think it's probably a critical thing to check if we assess how well a game went - something we might want to do down the line, especially since it makes an interesting second data point to compare with GM metrics - but it's not the answer to the question we're asking.

Make sense?

7 comments:

  1. Shouldn't 0 on Responsiveness be something like"unexpected actions threw off the DM, slowed down the game, and/or caused player friction?" It seems like "nothing unexpected happens" is the null value for responsiveness--nothing happened that causes the DM to need to respond. So shouldn't that get the median value of 1? Or am I not understanding the scale correctly?

    (In my head, I'm converting these values to -1,0,1, where the first is bad, the second is neutral/fine, and the 3rd is great).

    ReplyDelete
  2. I'd agree that 0 should be that the players tried something different and the GM reacted poorly to it. I agree that in most sessions the players will do something to throw you off but it's also possible that with time, A GM can know his players and their characters well enough that the game happens as expected. That to me would mean the GM is on his toes and has got things figured out which should count for something.

    ReplyDelete
  3. My thinking with 0 was that things went according to plan, comma, DAMMIT. Which is to say, whatever the players did, the GM shoehorned it back on track. Thus, a 0 may come up in a strong railroading situation, but it's not the ONLY situation. If the players do not initiate (for whatever reason) that merits a 0 as well. Admittedly, that doesn't directly reflect on the GM, so it may seem unfair, but I figure if the player's aren't initiating, there's a reason they think they can't/shouldn't.

    The other issue is that couching it in terms of "Staying on plan" makes it easier for a GM to self assess. "Did you squash player creativity?" (which is kind of the real question) will get an instinctive "Of course not!".

    There's also a subtle distinction between "expected" and "On Plan". I expect my players to wreck any plan I have, so I sort of skip 0 by virtue of not having plans that are designed to expect deviation. It's a cheat, but cheats are just a nastier term for best practices.

    All of this, of course, should be formally called out as part of the question, so excellent feedback.

    (Also, Joel, you're understanding the rating, but I'm trying to get away from the 0=bad, 2=good because on truth is that a game can have several 0's and still be an awesome game, so the point of zeros is not to criticize, but to raise questions of what was going on there.)

    ReplyDelete
  4. I've been thinking about these metrics as a measure of "Game Health" that a GM can track over time. Once you have some idea of how "healthy" your game is, you can try and work backwards to figure out how you're doing as a GM.

    By this logic, "Energy" is a perfectly acceptable metric. It gives you a sense of how "healthy" your game is. Sure, there's a lot that's out of your control, but that's true of all the metrics. Maybe you play on a weeknight, and all your players are tired because they're early risers, resulting in a low Energy score. Maybe you play on a weekend, and your players drink socially and unwind, resulting in a high Energy score. Like the other metrics, this just means that you have a different baseline depending on your situation.

    I also don't agree that "Energy" isn't actionable. As GMs, we have lots of tricks in our toolbag that can jazz up a room. Music selection, getting people up and moving around, etc. These are not easy things to do week after week, but they are tools we could be using.

    In the end, though, I do agree that "Energy" should be abandoned as a metric, but for different reasons. All the other metrics have relatively concrete measurements. "Unexpected player decision dramatically changed play" is a good example. It's straightforward and easy to remember after the game. "Energy" doesn't have such clear-cut definitions. The best you could do, I think, is something along these lines:

    0 - After the session, no-one seems particularly engaged in discussing what happened during the game, or what's likely to happen in future games.
    1 - After the session, half the group or less is engaged in discussing what happened during the game, or is looking forward to the next game.
    2 - After the session, everyone in the group is animatedly discussing what happened during the game, and speculating on what might happen next.

    The thing is, I would fold all of that under the "Engagement" metric. So I'm not sure what kind of definitions you would use to measure "Energy". Unless we can think some up, then I agree that it should be dropped.

    ReplyDelete
  5. Yes. Good sense.

    ReplyDelete
  6. @Pol perhaps I should clarify that I don't think energy is _directly_ actionable. Most of the things that solve the energy problem are also solving other problems (better descriptions, more engagement) so it's hard to isolate.

    That said, I'll reiterate that I totally wan tot track energy, but on a different sort of index. Player experience, perhaps?

    ReplyDelete

Note: Only a member of this blog may post a comment.