UserTesting is a commercial service that promises automated usability testing.
You might think of this as a series of task-oriented usability test sessions where you only need enter a number of tasks for a test that will take approximately 15 minutes, enter the demographics (age, income) and technographics (computer experience) for the user you'd like to target. UserTesting has a pool of paid testers who accept sessions, view your site with a floating layer that shows the tasks to complete while they record their voice, mouse clicks and movements, and page changes. Finally, the service asks the user to fill out a follow up survey. The cost is tiered by number of users per session.
Here's how it works:
1. You sign up for user testing, specifying the demographic profile of your target audience, how many users you want, and what tasks you want them to perform on your site. You can optionally enter more information about your user requirements, or enter names of specific UserTesting users you my have worked with in the past.
2. Usertesting notifies users. When users have accepted your test, the status change appears in your control panel. Completed tests can be viewed from this page or via an email notification.
3. Users record their screen and voice as they use your website, speaking their thoughts as they browse. You watch and listen to them use your site. Each user's session - mouse movements, clicks, keystrokes, and spoken comments - is saved as a Flash video for you to watch.
4. Finally you are able to read a summary the user enters at the end of hte session: What they liked, didn’t like, what would have caused them to leave your site.
Observations
The product seems targeted at companies that don't have a person on staff with experience doing user testing, who don't want to invest in time for preparing and facilitating test sessions, or who don't want to hire a usability consultant. Seems promising on the surface, but I think it's not without challenges.
A few things stand out from my brief demo. Without the physical presence at sessions, follow-up questions to specific tasks have to be scripted as part of the given script, and you cannot ask follow up questions while the test is in session. On the positive side, you can't lead tasks at all, but a facilitator with experience wouldn't be likely to fall into that trap. The forms make some good suggestions for typical tasks and follow up questions and how to word them.
You do not do the screening yourself. This has pros and cons. The pros are that they save you time. The cons are that you cannot screen people out after talking with them through your own screener, with your specific questions. The best that the sign up form offers you is a field to enter additional needs, but you're at the mercy of the service to follow through on your requests.
Another thing I'd like to have seen is the ability to write my own follow-up questionnaire rather than use the set provided, as well as offering some simple questions with ordinal value answers perhaps.
The best benefit on the surface is cost. The biggest challenge in my opinion will be in proper screening, and most importantly in the analysis. I have no doubt that the right company might be able to find value in a service like this if they choose the right scenarios, word tasks well, and are able to analyze the results and make suggestions for the right improvements.