How do you ask the right questions that give you information that is actually useful for you to make decisions? How do you make sure those questions are objective and unbiased? That is a question I try to address over and over again. This can be a difficult question for any marketer (especially me right now as I attempt to make a long multi path quiz) as there are infinite numbers of questions that you can ask, but only a few that you actually should. It’s also important to ask the right questions in the right way. Asking questions in the wrong way can increase bias and give you flaw results.
There are three main challenges that I have:
- Adding more questions means more friction to the user which means that you may have drop off, and less people actually taking your survey or quiz.
- Adding too few questions may mean that you do not have enough information to make the right decision based on the data.
- Adding the right questions and asking the wrong way can create bias in my results, making my data essentially worthless.
So, how do you know whether you have too many or too few questions in your survey or quiz? How do you know whether your questions are objective? Let’s start with a simple 5 “A”’s framework that you can use to determine just the right amount and the right questions for your user to improve your quiz response rate right now (and have those answers be relevant to actually solve a problem).
Acknowledge and understand your user. What pain points is your solution trying to solve? What insight do you want to gain from your customer? What motivates your user?
It’s a problem if you do not know which problem you are trying to solve. Too quickly we can jump into the what we need to do without full understanding what problem we are trying to solve.
For Ascend, we are a personal finance firm that wants to understand which debt elimination method that our customer should use. There are many such as debt settlement, debt management, bankruptcy and debt consolidation and there’s a right option for each person, so how do we make sure that we give the right example.
Your customer has several pain points. Get specific about which problem you are going to solve. If you try to solve all of his/her problems at once, you will solve none of their problems.
This section is important because asking the right number of questions and the right content of those questions can be the most important attribute of a successful quiz.
What questions do you ask?
The goal of the questions you ask should be to create an objective response from the user. Without further ado, let’s talk about Response Bias. Wikipedia defines Response Bias as, “a general term for a wide range of tendencies for participants to respond inaccurately or falsely to questions. These biases are prevalent in research involving participant self-report, such as structured interviews or surveys. Response biases can have a large impact on the validity of questionnaires or surveys.” Let’s get into some of the response biases:
Acquiescence bias: Acquiescence bias, which is also referred to as “yea-saying”, is a category of response bias in which respondents to a quiz or survey have a tendency to agree with all the questions in a measure.
An example of this:
The bias is that it is often harder to say ‘no’ than ‘yes’, so a better way to ask this question is as follows:
Demand characteristics: Demand characteristics refer to a type of response bias where participants alter their response or behavior simply because they are part of a survey or experiment.
Extreme responding: Extreme responding is a form of response bias in the quiz or survey that drives respondents to only select the most extreme options or answers available.
Question order bias: Question order bias, or “order effects bias”, is a type of response bias in a quiz or survey where a respondent may react differently to questions based on the order in which questions appear in a survey or interview.
Social desirability bias: Social desirability bias is a type of response bias in a quiz or survey that influences a participant to deny undesirable traits, and ascribe to themselves traits that are socially desirable.
How many questions do you ask?
Only the amount that solves the problem you are trying to solve. In my personal experience, the more questions that you ask the greater chance that you have that the user will not continue. The less questions the better as long as it’s addressing the problem you are trying to solve.
Okay, this “A” may be more of a stretch, but the idea here is how do you create the right ambiance for the user to answer the questions. Is your survey one page with one hundred questions or one hundred pages with one question?
I may want to write an entire article just on this piece as this can be extremely important to delight the customer and eliminate frustration.
How do you know whether your quiz is actually working? You need to analyze the results and iterate accordingly. Sometimes we are too confident that something is working just because we put it together and it made sense at the time. Let’s have humility and analyze the results.
Interact has a great analytics function to show you a high level metric of your funnel conversion.
I really like the Questions and Answers tab on Interact that shows whether individuals are dropping off at any point in the quiz. This will give you insights as to whether there are question that are not going well with your user or you’re potentially asking too many questions.
I may have left the most important thing for last, and that is how do you appreciate your customers for the time that he/she just devoted to give you valuable feedback about his/her pain point?
I receive potentially 10-20 feedback forms from various banks each month about how to improve the product. My guess is that you do as well. Do I ever answer any of them? Almost never. Why? Because I don’t believe that any individual will actually read my message and I certainly won’t get a response. If my time is not valuable to the requester, why would I respond?
This is the exact same thing for survey and quizzes. How do you appreciate those who use the quiz or survey? This can be as little as saying “Thank you” at the end or even reaching out to the individual who took the quiz. Little things go a long way. I mean it.
Hopefully this gave you a simple framework to ask relevant questions to improve your quiz response rates right now. It’s not easy to make great quizzes, but it’s imperative to understand your user to make great products and solutions for them.