Beta Testing
Posted May 19th, 2004 @ 02:09pm by Erik J. Barzeski
Primarily for myself, here are some rules our recent beta test taught me, in no particular order (but numbered for my own sanity):
- Don't do open betas. People just like to use free software, and most of the reported bugs are worthless. "It crashed when I clicked on the toolbar."
- Carefully select beta testers, and don't just let anyone in who asks. Make them confirm repeatedly that they'll contribute and pare out those who don't. Make sure that beta testers know that they're testing to find bugs, not to voice support for their pet features.
- Beta tests must run at least one full month. Bring in new testers in the last week, and extend the deadline a week at a time until all major bugs have been squashed.
- Ship at least four releases to beta testers, once a week or so. Any faster than that and some won't bother to upgrade.
- Let beta testers have access to the bug logging software. Don't make it too hard to file a bug: request the minimum information. Version, steps to reproduce, crash log, and a screenshot if possible, along with a descriptive title.
- Give free copies to anyone who's left at the end of the beta test. If they didn't do anything, they should have been given the boot earlier.
- Make it clear at the beginning of the trial that beta testers can be booted at any time for any reason. Help out or get out.
- Stick to your guns. Adding features during beta testing is the fastest way to delay product release, introduce new bugs, and cause problems.
- Provide a list of all new bug fixes or changes with each release. Provide a list of the functionality that you wish people to bang on in particular for the current cycle.
Yep.
Donate Life
Posted 19 May 2004 at 3:39pm #
An excellent summary, thanks. It's this kind of "obvious" stuff that often bears repeating!
Posted 19 May 2004 at 6:19pm #
yep, that looks like a pretty good list. I've seen too many beta programs which had a complete lack of communication about what bugs have been fixed, what areas need to be examined, or what has changed between versions. And in some cases, testers have been given no instruction on how to file reports.
by the way, is there any preferred way to report bugs in PulpFiction, other than just sending email? 🙂
Posted 20 May 2004 at 10:08am #
NSLog Beta Lessons
From the good people at Freshly Squeezed Software, a very good collection of Beta Testing Do's and Dont's. This intrests...
Posted 21 May 2004 at 7:28am #
Beta Testing
Yep, these beta testing tips are going to be quite useful for me soon. 🙂