Listen Up: User Research on a Shoestring

Last time, I wrote about why you need to do user-centered research for your website.

This post is about how to do it. It doesn’t require a lot of money, but it will require a lot of time. (Sorry, there’s no short cut – it will either cost you time or money.)

Interview: Get a general sense of the condition of your current site by talking informally to a few volunteer leaders, senior staff, and some members.  Ask general questions about their experience of the site. Ask them to perform a few basic transactions or tasks on the site and watch what happens.

Study: If your site logs are any good, study them. Look for trends in visitors and the time they spend on the site. What are the top 20 or 30 content pages? What transactions are being used most often? What are the most common inbound search terms? Are there any patterns to the error messages? This is hard data that will ground you in reality.

Prioritize: Presumably, your senior staff is already on the same page with regard to business priorities. If not, stop NOW and do that first! If so, then you need to prioritize how the site will fulfill those business priorities. For example, if your membership department is targeting young members, how will your site recruit new, younger members? If your business is using education to generate revenue, how will the site deliver knockout online courses?

You’ll end up with between one and three dozen site objectives. Have your staff put each site and business objective into one of three categories: Must have, need to have, and nice to have. Be careful; your team must be ruthless in doing this. Often, they will put everything in the top two categories in an effort to reach consensus. Bad idea. If too many things are a “must have,” your site will (continue to) be a bloody mess.

Force your team to reserve only a handful of things for “must have.” And make them put an equal number of things in the lowest priority. If you call the categories “critical,” “very important,” and “important,” no one will feel invalidated, and your staff may find it easier to do this.

Listen: Go to your members and talk to them about your site. There are dozens of ways to do this. We recruited five focus groups, segmenting them by age.

We limited committee and volunteer leaders to only 25% of the focus group members, because we want the site to be of interest beyond our core leaders. This took some work, and some time. It’s worth it. Don’t go for the usual suspects. If you do, you’ll get the answers you’ve always gotten.

We offered a $200 cash incentive for coming to the session. For medical students and practicing physicians driving for more than one hour, we paid for their mileage. This helped make our groups more diverse.
At the start of the session, they filled out a short questionnaire about how they use the Internet professionally and personally. The questionnaire had nothing to do with our site. Then we talked with each group about our site and how they use it. Some of the answers were quite humbling.

Then we did a paper exercise with them. We created two big sheets of paper with about three dozen boxes representing major content areas of the home page that we guessed they would be interested in. Some were old ideas, some new. We gave them scissors and glue sticks, and had them arrange the boxes in order of priority on another ledger size sheet of paper. The question they were answering was: What would they want on the new site?

Each session lasted about two hours. When it was all over, we compared our staff’s priorities to their priorities. Most times, we had a close match. On others, we were miles apart. Almost always, when there was a difference, we followed what the members said.

Design: Now you’re ready to draw your site map. From that, you can develop your wireframes. There’s both art and science in this. My key advice is, don’t organize your site by departments. No one knows your org chart but you, and no one cares. Organize it by your users’ goals, because that’s how users think when they go online.

Many designers go through a complex persona development exercise. These are valuable, if you know how to do them. However, most of us require outside expertise to develop them, and that requires money. If you have it, it’s worth considering. If not, thinking strictly and rigorously from the vantage point of your users’ goals will get you almost all the way there.

Test: Show your wireframes to at least a dozen members in a one-on-one setting. Ask them to accomplish at least two dozen different tasks, such as renewing their membership; learning about the association’s position on a legislative issue; taking a course; registering for a meeting; finding another member; reading a news release; finding a staff person.

At least 60% of your tasks should be successfully completed on the first click. If another 10% to 20% find it on the second click, it’s not a huge problem; it means you have a little tweaking to do.  If more than 10% to 15% of the tasks are utter failures, step back and take a look at where they failed. There’s probably a pattern in there. Go back and look at your focus group notes; you probably didn’t really listen to what they said.

Stakeholder management: I took the site map, wireframes and the usability test results to my organization’s major stakeholders. I showed them how the data proved that the material they cared about was easily found by our members, despite the new content structure. The results were clear, and no one could argue against success.

If you do this right, you will get three results. Some of your ideas will be validated. Others will be proven dead wrong. And, you will learn something absolutely new about you and your members. That’s really, really cool.

The ability to learn these new and exciting things entirely depends on you having open ears and an open mind.

And that’s doesn’t cost a single penny.

Recommended reading: The User Is Always Right, by Steve Mulder