Surveys are often poorly designed

Surveys are a valuable tool to understand topics you might otherwise be unable to explore well. Best survey design is done by a specialist in this area, someone who properly understands the statistics and psychology of surveys. However, while not everyone can have a PhD in this area, anyone can still level up their skills, collect better data, and improve the experience for survey recipients.

Too often, I see survey designers just throw together a bunch of questions that broadly relate to the topic, give each a 1 – 5 score and a ‘Why did you choose this score’ question to try to include quantitative and qualitative measures.

This is noble but results in:

• Questions that turn out to be ambiguous
• Questions that yield useless data
• Extended survey length leading to fatigue and higher abandonment from survey recipients

I do see survey designers doing great things like test that the survey is functional and reduce ambiguity in their questions so I won’t talk here about that.

My trick is a bit more subtle but I’ve seen dramatic results – and people don’t think of it because they don’t think like a data person.

Make a dummy data set

Making a dummy (fake) data set based on the questions you want to ask helps the designer and business visualise if they will actually get useful results or not. Very often a question seems valuable but looking at dummy results makes you realise it was actually biased or too rigid.

For example, I might be building a cereal start-up but am worried about my market size.

I develop a survey asking a key yes/no question: Do you eat breakfast?

To help explore the usefulness of this question, I make up a potential result to help discuss.

Make sure you emphasise this is dummy data otherwise people are inclined to just believe things in charts.

This can help the discussion, because I start asking What if this were true?

I might say, well if 74% ate breakfast I might have a business – but I really need to know how often they eat breakfast to know for sure.

This allows you to refine the question to something like: How many days per week do you eat breakfast? (with a 0-7 score).

Now, when I run the survey, it will better help me mitigate viability risks by providing a better qualitative understanding of the Total Available Market.

This isn’t a silver bullet, but this helps build What If? scenarios with survey designers and the businesses in an engaging way.

If you are finding your surveys aren’t hitting the mark in actually being useful, try this approach and see your data quality and business collaboration improve.