Do usability expert evaluation and test provide novel and useful data for game development?

Laitinen asks whether expert evaluation tests provide novel and useful data for game development, and in particular whether it makes a difference whether the experts are also gamers.

In his case study Shadowgrounds was evaluated by usability experts, and then tested by game developers. The team of experts created a report of issues such as the following,

No feedback is given if the player cannot pick an item.

In my opinion this kind of issue might not be picked up by either designers or quality assurance staff. They would both be conditioned to recognising that when their inventory is full no more items could be collected.

In the usability study, players were asked to think out loud, and were sometimes questioned or tasked by the usability instructor. Again, I notice that this does not represent a natural environment within which to play games, but in this case perhaps it doesn't matter. The tests were evaluating interface and game play issues only, so perhaps it's appropriate that they were conducted as quite specific "tests" - to test if the player understood how to move the character and interact with the other entitities in the world in the manner intended by the designers. Laitinen does note that it was sometimes not reasonable for the interviewer and interviewee to discuss the game during action sequences due to the level of involvement required to play.

The results are very interesting. 43% of issues found in the expert evaluation and usability testing were unknown to the developers, and 74% of them would have gone unnoticed without the evaluation and testing. It's not specified whether the developers had a QA team or not though. I think most developers are unaware of the myriad issues a regular QA team find, so perhaps these statistics are not so exciting (other than to further emphasise the case for having QA!). Most of the issues found were not programming errors, so were probably rather design issues. QA tends to be more focussed on finding bugs than this softer design issues. Furthermore, while the developers only rated these issues are being quite relevant, most of them were addressed and the suggestions given by the experts were generally considered useful.

Interestingly the data suggests that it did not make a difference whether the experts were gamers or not, all were usability specialists and were able to make the same quality of judgements. However, Laitinen does suggest that if they had more experience of playing this specific game the results might have been even better.

Laitinen, Sauli. "Do usability expert evaluation and test provide novel and useful data for game development?". (Journal of Usability Studies, 2.1, February 2006), pp. 64-75

Shadowgrounds. Frozenbyte, Inc. (Plan 1, Meridian4, Valve, 2005)