Low-Fi Usability Testing

There's an intersting article about software (not games) usability testing which discusses frequent, rapid and low-cost testing. Citing Steve Krug's book Don't Make Me Think (from which you can read Chapter 9: Usability Testing on 10 cents a day) and Jakob Nielsen's paper "Usability Engineering at a Discount" and "Why You Only Need to Test With 5 Users", the author has the following pertinent advice:

• When should I test? Ideally, once per month


• How many users do I need? Three or four max.

And one of the comments suggests this,

"doing two tests with 3 users each finds out more bugs than doing one test with 6 users, or 8"

This sentiment is similar to Nielsen's "most projects require multiple rounds of testing" with "representative users"

Additionally he advises beginning with qualitative (insight) tests using 5 users before moving on to quantitative (statistics) testing with 20.

However, these observations are based on data collated from web usability tests, so an important question for video game usability is whether these number are appropriate. It would be useful to conduct the same kind of analysis as Nielsen in order to devise product-specific heuristics.

I would suggest taking 20 as a starting figure for a quantitative study, but continuously compute the SD through the test to calculate the current margin of error. In order to do this it's necessary to have a mean value, which requires some data to be collected so we'd have to assume 20 users initially, but recalculate the mean and margin of error after each test.