A Song of User Research

Surveys and not doing them wrong

Part 2: The Double Negative Edition

Welcome to Part 2 of my 4 part series on User Research. Check out Part 1 on user interviews if you’re interested!

Courtesy of Suat

The beginning

Put your hands up if you’ve heard of surveys. Great, now put them down.

Now put your hands up if you’re here for some fun. Everybody? Okay great.

That was your first lesson.

Anyway, today we will talk about UX surveys and discuss some guidelines to help you avoid writing up a biased or ineffective one. Survey writing is not as easy as one might think, and dispersing a poorly effective survey can be a huge waste of time, money, patience, and valuable user feedback.

So with that, let’s vroom vroom (go).

A quick briefing on surveys

Surveys are a series of questions designed to be answered by people who represent your user personas, and are a highly common research method employed by UX researchers/designers. Surveys help to provide information from users that one may not get otherwise from user interviews, and are therefore highly important to your UX research.

In general, survey data is used in conjunction with user interview data (or any other type or research data) because it has certain benefits over user interviews. These benefits are:

1. The ability to reach a larger number of users
2. Being effective in gathering large amounts of quantitative data
3. Reflecting or showing disparities between the quantitative and qualitative data

Surveys are more often focused toward quantitative data over qualitative, but that does not mean that qualitative needs to be excluded from the questionnaire entirely. Surveys are able to accommodate short answer responses as well, but the amount of which should be limited (we’ll speak to that a bit further in this article). Moreover, surveys aren’t necessarily limited to a particular method for dispersal. They can be done both in person or dispersed out into a population of your user personas via email, mail, or on social platforms.

Surveys are important, do them correctly!

It’s me! (Source: Kit8)

Like the title of this section says, surveys need to be written correctly lest you waste your time and are terribly saddened. There are a number of pitfalls that surveys can fall prey to, and in my earlier designer days I had certainly stepped right onto them and fallen in. I had the boo boos to prove it.

So, to help you all along, I will lay out some rules that hopefully prevent you from having as many survey boo boos as I did. Let’s get to it.

Important survey rules:

1. Orient yourself to the goal of the survey

Just like you would do if you were to create a user interview (or even further back- a research plan), be sure to know what the goals of the survey are before you begin haphazardly writing the questionnaire. This will help you avoid straying from the path and asking irrelevant or unnecessary questions.

2. Limit the use of qualitative responses

… Unless you want a qualitative survey. A major benefit of surveys is that they can reach a large number of users. This is why most surveys employ the use of quantitative questions, because analyzing a huge amount of short answer responses would suck. However, sometimes short answer questions are okay, but use them sparingly. This will benefit both you and the users because:

1. They don’t have to type up most of their answers
2. 2. You don’t have to read 100 essays.

If you were to do a qualitative survey, I would suggest keeping the amount of questions low. Overwhelming the users with short or long answer questions can be.. Well.. overwhelming. Ultimately, the best qualitative data comes from speaking to users directly. So you can leave it up to user interviews to get the best qualitative insights, and surveys to get the best quantitative insights.

3. Use screener questions

This is especially important for dispersing a survey into the ether of a social platform. If you don’t add questions to vet people that are not representative of your user base, then your data isn’t accurate. Environmental scientists don’t study bears in Alaska to learn about shark migration in the Atlantic. (although if they found a connection there it’d be cool as s***).

• Here is a quick example: If a researcher for a ride share app wants to learn why it’s users who are aged 50 years or older often choose not to rate their drivers, they would need to make sure that only users of their rideshare app who are older than 50 years of age would respond to their survey. So some really obvious questions would be:

1. “Are you 50 years or older?”
2. “Do you use ____ app for ridesharing?”

It’s as simple as it is important. If you don’t include screener questions then you have no idea if your survey data actually represents your users.

4. Construct a logical survey

“Construct a logical survey” probably sounds way too obvious, but you might be surprised to find out how many surveys aren’t always constructed logically, or fairly for that matter. What I mean to say is this- sometimes surveys are written in a way that feels either out of order, or (unintentionally) uses unfairly worded questions. Here are a couple of ways to avoid that:

1. Group similar questions together
2. Write the questions in a way that is easily understood by the participants. Basically, avoid using language that might be esoteric to users AND don’t use double negative questions. An example of a double negative question is this: “Is the website not easy to use unless you use a search bar?” Bleh.
3. Use a balanced rating scale for questions. As in, don’t include more positive answers than negative. The best example is using 2 positive answers, 1 neutral answer, and 2 negative answers for a total of 5 possible answers the user can choose from.

5. Test your survey before dispersing it

This is extremely important to the integrity of your survey, and can be done at several levels:

1. Just like writing your will, proofread your work before you declare it done. Please carefully reread your questions to see if they make sense.
2. Better yet, get someone else (perhaps an impartial UX colleague) to read over or even take the survey as a quick test.
3. Furthermore, you can test it with a small sample size of users to see possible trends displaying confusion with questions, not enough (or the correct) choices for answers, or a gauge for thoroughness in achieving your goals.

I’ve personally experienced embarrassment for having put out a messy survey without proofreading it or showing it to a colleague to ensure it made sense. You don’t want that feeling.

6. Explain yourself

This one has a lot to do with research ethics. When you are giving anyone a survey, you must be transparent with what the information they provide will be used for. At the top of each survey, explain these things clearly:

1. The purpose of the survey
2. Who the research is being done for
3. How will the results be used
4. Whether or not the survey is anonymous

Be ethical, therefore be transparent. Sometimes people are very excited to participate in your research, and sometimes people are very wary. However, both kinds of people deserve to feel certain in knowing what they’re in for.

Wrapping up

(Source)

Like I mentioned earlier, surveys are very common research methods that provide integral data for your project. However, precisely because they are commonly used, surveys must be learned to be done correctly to avoid incorrect or useless user insights.

Here is a quick summary of those rules I gave:

1. Orient yourself to the goal of the survey
2. Limit the use of qualitative responses
3. Use screener questions
4. Construct a logical survey
5. Test your survey before dispersing it
6. Explain yourself

I don’t intend for surveys to sound like they are difficult beasts to tame, because they are far from that. Like many things, though, they take time to understand. But you’ve got this. You’re a UXer, by golly.

Thank you for reading!

This article was written based on research, experience, and the insights that I’ve gained throughout my UX journey. Please feel open to discussing any thoughts, questions, or conflicts that you may have and I’d be happy to learn more.

Also, I encourage you to add me on LinkedIn. Just be sure to say hi let me know where you’re from!

My LinkedIn: https://www.linkedin.com/in/samueljaklich/