Heuristic Evaluation for Playability

I've just added Desurvire's professional site, Behavioristics to the links section, and have read a paper she co-authored with Caplan and Toth, called Heuristic Evaluation for Playability

In this paper the authors present a system of design guidelines called Heuristic Evaluation for Playability (HEP), which they claim is particularly useful for evaluating pre-production prototypes and for addressing issues of game story and interface.

They catagorise their 43 different heuristics into game play (16), game story (8), game mechanics (7) and game usability (12). Interesting for me is that game play is defined in terms of mastery, of beating the game. There is no consideration given to subversive or emergent play which takes place beyond the intention of the designers.

Reading this piece makes me wonder about the best way to evaluate game usability. The following quotation strikes me as a good case study:

Each session was organized as a one-on-one evaluation session, in an environment similar to the one where they would actually play the game. Participants were given instructions to begin the game, asked to think aloud, and asked several probing questions while using the prototype.

I don't know of any game where I have to think out loud and am probed while I play. Surely this has an effect on how I play, or how I perceive the game? A less intrusive approach might be to use automated data gathering techniques inside the game to monitor the player's progress. At some stage it would be necessary to ask for their subjective evaluation of affect, but I wonder if this couldn't be inferred from their gameplay itself? Given the option to continue playing or stop early with no negative consequences it would be fairly easy to recognise if the player was having fun or not. The main problem with this thinking out loud and probing questions is that it disrupts a sense of immersion in the game. This would have significant consequences in games which require concentration, or which rely on mood as their aesthetic pleasure.

I wonder if HEP is a ludological / narratological analysis at the expense of the carnal pleasures of game play?

In conclusion they state,

The user studies findings highlighted issues specific to the game; boredom, challenge and pace level, as well as terminology. These issues were not found through HEP, whose benefit was in ensuring general game principles.

This is an important acknowledgement of the appropriateness of HEP.

Desurvire, Heather; Caplan, Martin; Toth, Jozsef, A. Using Heuristics to Evaluate the Playability of Games. (2004)