'Lo about 2005 I started doing usability testing. I was working on a major redesign and replatforming of continental.com and we tested every interface multiple times, in cities across the U.S., before handing off the requirements to the development team. (This was also a different age when things were far less agile than today.)
Since there were only three of us to do all the UX design and usability testing, this was a formidable task which was only made more challenging by the need to recruit from and travel to the various cities. Sometimes I conducted testing, sometimes I observed and took meticulous notes from behind a one-way mirror. One time the one-way mirror was inside a room that I suspect was once a very small broom closet. My co-worker had to leave in the middle of a session because she became claustrophobic.
Usability testing has come a long way in 7 years.
Today, there are no more broom closets for us. The vast majority of usability testing we conduct at Normal Modes consists of:
- User interviews onsite at our client's facility - almost exclusively B2B clients testing highly complex, very specialized software applications with participants who leave their desks to join us for an hour in a conference room.
- Remote usability testing - almost exclusively B2C clients testing website functionality. Good examples of this type of testing are our Republican Candidate Website Usability Study and Weight Loss Website Usability Study, both of which were conducted with remote usability testing.
- Morae usability testing software enables us to more easily capture video and integrate observer notes. The ubiquity of webcams allows us to move away from a separate video cam and rely on picture-in-picture functionality.
- GoToMeeting, and similar products like WebEx, allow anyone with average bandwidth to join a conference call. Last year Citrix, the company behind GoToMeeting, rolled out an upgrade that allows us to conduct video conferencing over residential-quality internet connections.
I'm a huge fan of remote usability testing.
Here are six reasons why:
- Ease of recruiting & convenience for participants - Convincing a potential participant to schlep to our Houston offices - even though they're very conveniently located in central Houston - can be a challenge. Houston's a big city (the fourth largest in the U.S.), so a suburban participant could easily spend an hour commuting each direction into the city for the study. With remote usability testing, they can participate from the comfort of their home or office, with no commute time. That fact decreases the amount of time it takes to recruit qualified participants and eases their accumulated burden.
- Geographic diversity in participants - In a recent study we recruited participants from across the U.S., with participants ranging from a guy in suburban Boston to a lady in remote Alaska. Because we test from 7a-9p CT, we can leverage time zone differences to more easily accomodate participants. And because we have all these time zone differences, there's no afternoon lag - there's always a "peak desirability time" available in one of the time zones.
- Lower no-show rate - In the past, when I conducted B2C in-person usability tests, at least 2-3 out of 15 participants didn't show. With remote usability testing, we usually have only one no-show, and that person often reschedules. Again, the convenience of participating from one's home or office reduces the amount of effort required to participate in the study.
- Lower overall costs - Not having to rent a room equipped with a one-way window shaves about $3,000 off the usability testing budget. And not having to travel to another city - with the associated air travel, car rental, hotels, per diem and additional billable hours - can save considerably more.
- All the access of traditional testing - Some people have criticized remote testing because they're unable to see the participant's face and, as a result, miss important non-verbal cues found in facial expressions and body language that indicate user confusion, gaze direction, and gestures. Again, with the ubiquity of webcams and availability of video conferencing software, we don't find this to be a problem. In fact, we find that people tend to behave more naturally because they don't feel like they're being watched and because they're less likely to try to develop a strong rapport with the moderator. And because the participants are in their natural environment without strangers in their immediate physical space, they can be more focused on the tasks at hand.
- A wealth of additional information about user environments - Because we can see the user's desktop, we're able to gather additional information about actual working environments - rather than the sterile, uniform environment we provide in labs. We've had participants who literally had a half screen of additional toolbars on their browser, which considerably affects how they find and interact with websites. We've worked with older participants who, due to vision problems, used very low screen resolutions resulting in considerable horizontal scrolling and difficulties interacting with webpages. We've also had participants who, during a decision making task, opened up their own files in order to find the data needed to make the decision. This is how people actually use websites.
Sure, about once a session we encounter technical hiccups, usually with older participants and always related to GoToMeeting. (WebEx is no better.) Bandwidth hasn't really been an issue in our studies; we always pipe in video from participants only. We've only ever had one participant who didn't have sufficient bandwidth; she was in her 70's and didn't have a webcam either. In that case, we paid her the incentive and quickly found a new participant. Easy peasy.
Of course, not every situation is suitable for a remote usability study.
Studies that aren't appropriate for usability testing tend to be on applications where there are technical limitations (usually B2B situation with non-public, proprietary databases) or where the user's proximity to the application is a constraint. Mobile devices or kiosk applications, for example, need to be studied in person.
Is that really all there is to know?
Well, you've got to know how to plan and conduct a usability test, too. Don't forget that. We can help there. Our usability testing workshops cover both how to design a usability study (test plan, screener, and moderator's guide) as well as the technical setup for conducting any type of usability test.