How to write unbiased survey questions

Surveying is one of the most common research methods. Whether you’re a qualitative researcher or a market researcher, you must be able to write unbiased questions.

Writing unbiased survey questions is easier said than done, but don’t worry, there are simple things you can do to ensure you remain objective and keep your questions unbiased.

In “Think Like a UX Researcher” by David Travis and Philip Hodgson, they call bias one of the “seven deadly sins of UX Research.” Why? Because your bias can skew your data to the point of being unusable. The scariest thing is that you may not realize your bias until you discover that your proposed solutions are ineffective.

Your bias only has to surface at one stage of your business, marketing, or UX research for an entire study to go off course. When conducting qualitative and quantitative research, the questions you ask are crucial, and what’s even more important is how you ask them.

What is a biased survey question?

Being biased means, you have an inclination towards one thing over another. This inherent prejudice can get in the way of conducting an objective and neutral research study. I use the word inherent because it is impossible not to have feelings about a topic, especially when you’re conducting an in-depth research project. What a good researcher can do is remove their bias by carefully wording their questions, whether they are for an interview or survey.

Here’s an example of a biased survey question:

How much did you enjoy the cup of coffee?

It seems like a simple enough question, right? Imagine a big coffee chain was conducting a survey, and this was one of the questions to find out the customer satisfaction score. The participants of the survey are automatically thinking about how much they enjoyed the coffee. That’s where the bias comes into play; the creator of the survey is assuming the participants enjoyed the coffee and even leading them to believe they enjoyed it.

A better way to phrase the survey question:

How would you rate the cup of coffee?

This phrasing removes bias and simply asks the participant to rate the coffee without leading them to any conclusions.

Things to avoid when writing survey questions

There are several ways for your bias to throw off your research study. The list below has a few of the more common mistakes researchers make when writing product survey questions.

1 - Assumptions/loaded questions

One of the most challenging things about remaining unbiased is removing any assumptions you have about your participants. Some refer to assumptive questions as loaded questions. It’s when you phrase your survey question in a way that assumes something specific about your participant.

For example, the question below assumes the user drinks tea:

What is your favorite tea?

For loaded questions, it’s important to consider the context and even the order of the questions you’re asking. You can’t always correct an assumption with phrasing, sometimes, it’s more about the gathering of your information. The question above would not be a loaded question if you previously confirmed that the participant drinks tea.

2 - Leading questions

A leading question uses phrasing that suggests an emotion, experience, or any outcome or thought that the user may have had. In the coffee example above, participants end up thinking about how much they enjoyed the coffee instead of considering their own subjective experience. Here’s another example:

❌ Biased:

How amazing was your viewing experience at our theatre?

✅ Unbiased:

How would you describe your experience at our theatre?

In the biased question, participants think about how amazing their experience was instead of considering how they truly felt about their experience. A good way to avoid leading participants is to avoid adjectives, don’t describe how they may have felt in your question.

3 - Jargon in question

Every industry has its own jargon; these are words that aren’t typically used in everyday language. But, when you’re conducting in-depth research, these terms may seem common to you when they’re not familiar to others. Using jargon in your survey questions can cause confusion, even if the participants are well-versed in the problem space you’re exploring.

❌ Biased:

How would you improve your scrum meetings?

✅ Unbiased:

How would you improve your team meetings?

The participant may not understand the word scrum or may think of a specific type of scrum meeting. It’s better to replace the jargon with a more common word.

4 - Negativity in questions

There are a few reasons why you should avoid negative phrasing when writing survey questions; they can be confusing, they’re often leading, and it simply gives your survey a negative vibe.

❌ Biased:

This experience did not meet my expectations (T or F)

✅ Unbiased:

This experience met my expectations (T or F)

The reason negativity is confusing is because the answer often means the opposite. If your participant answers True to the biased question above, they are saying the experience did not meet their expectations. If they answer False, they are saying the experience did meet their expectations. They’re basically being forced to answer yes or no when they mean the opposite, and that causes unnecessary confusion.

5 - Absolute questions

When you include absolutes in your survey questions, like “always” or “never,” you’re making your participant choose one or the other, and usually, both answers will be incorrect when it comes to the participants’ actual activity, thoughts, feelings, or actions.

❌ Biased:

Do you always use our website for research?

✅ Unbiased:

When do you use our website for research?

The problem with the biased version of this question is that the answer will likely be no, and if the participant answers yes, then there’s a good chance you have false data because no one really does one specific thing 100% of the time. It’s more valuable to discover when and why your participant uses your website.

Remain unbiased through all stages of research

So now that you know how to write unbiased survey questions, it’s important to remember to remain unbiased throughout your research study, especially when it comes to analyzing and synthesizing data. Cognitive bias can surface at any point during your user research, whether you’re conducting testing using usability testing tools or defining your information architecture.

1 - Confirmation bias

One type of cognitive bias is when you have a belief, theory, or hypothesis, and it is tempting to interpret your data to fit what you believed all along. This is confirmation bias; avoid it at all costs. It can lead you to ignore important data simply because it does not correlate with your initial assumptions.

There are a couple of good ways to avoid confirmation bias. First, approach your data with the intention of proving yourself wrong. Secondly, work with your team and stakeholders (but keep in mind that they may struggle with confirmation bias, too).

2 - Implicit bias

It’s natural for you to be more receptive to certain types of things, but you likely don’t even realize it, which is what makes implicit bias so dangerous for researchers. For example, you may overlook a relevant participant because their name is unfamiliar to you due to cultural and language differences.

You can avoid implicit bias by constantly reminding yourself that as a user researcher, you are an advocate for your users and always empathetic and compassionate; they are significant to you because of your screener, not because of their name.

The importance of writing unbiased survey questions

Keep in mind that your bias is not some evil entity that is trying to ruin your research study. You’re not being biased on purpose, but being unbiased is intentional.

If you write biased survey questions, the results of your survey will be unreliable, it’s as simple as that. And with unreliable data comes ineffective solutions, which leads to wasted time, money, and resources. Writing unbiased survey questions requires a lot of effort upfront, but if you want to generate valid data, it’s a must.

So, when it comes to creating your own survey, avoid the types of questions listed above and remember to keep your bias in check throughout your entire research study.