In an earlier post titled, “5 Simple Ways to Improve Your Quiz Response Rate Right Now”, I shared a simple 5 step framework that can help you achieve more and better quiz responses. To recap, the 5 A’s to increase your response rate are:
As stated earlier, I realized that each of these A’s could have it’s own article, so I wanted to go through an in-depth analysis of each of the steps in the framework. Today, I will walk through how to “Ask” the right questions.
Before you begin, make sure that you have acknowledged who your customer is and what pain point you are trying to solve for your customer. For Ascend, the pain point that we have been solving recently is that there are many options when someone is in extreme debt, but there are few unbiased and mathematical solutions that can give our customers the right path forward.
What questions do you ask?
The goal of the questions you ask should be to get a response from the user that provides insight into how to solve the user’s pain point, while avoiding all response bias.
What is Response Bias? Wikipedia defines Response Bias as a “…term for a wide range of tendencies for participants to respond inaccurately or falsely to questions. These biases are prevalent in research involving participant self-report, such as structured interviews or surveys. Response biases can have a large impact on the validity of questionnaires or surveys.”
There are many types of response bias, but the more prevalent are:
- Acquiescence bias
- Demand characteristics
- Extreme responding
Question order bias
- Social desirability bias
This is the area of response bias in which respondents to a quiz or survey have a tendency to agree with or say yes to all the questions in a measure.
I added acquiescence bias as the first to address because I find myself (someone who has loved to build quizzes for many years) falling into this trap over and over again.
Even after I had written the article, “5 Simple Ways to Improve Your Quiz Response Rate Right Now”, I fell into the trap. The story is that I was building a longer question quiz flow for our Chapter 13 payment plan calculator.
As I was building the quiz, I wanted to understand where the consumers were at in the bankruptcy process, so I asked the following question:
The results looked so biased that thankfully I was prompted to ask the question differently just one day after asking the initial question. What did I add that was a better question?
The initial results from the biased questions indicated that most people had an attorney. The results of the second question showed almost the exact opposite results:
Acquiescence bias is such a common trap that I will ask you right now to look at your quizzes and see how often this bias has happened to you as well?
In this response bias, participants alter their response or behavior because they are part of a survey or experiment.
Let’s go through an example:
You are conducting a quiz with two parts: 1) Reading and answering quiz questions. 2) Listening to quiz questions read to you and answering them. The quiz giver somewhat dismisses the area that the participants need to listen to the question and lets slip that the participants should pay close attention to the reading quiz portion. The participants thus focus on the reading portion and it negates the results of the quiz.
Extreme responding is a form of response bias in the quiz or survey that drives respondents to only select the most extreme options or answers available.
Most of us have received a quiz that asked us to rank something on a scale of 1 to 10. Extreme responding is the bias where quiz takers are more likely to answer the question as if they like it or hate it. Let’s look at an example.
Question-order bias, or “order effects bias”, is a type of response bias in a quiz or survey where a respondent may react differently to questions based on the order in which questions appear in a survey or interview.
The order of your quiz questions may matter more than we think. One of the better known examples of question-order bias is from Shanto lyengar & Donald Kinder in 1987 where they asked the following questions about:
- The most important problem facing the nation.
- Whether the individual approves or disapproves of the way the president is handling his job as president.
In this study, it was found that many of the people taking this study answered the approval question based on the previous question of the most important problem facing the nation.
I address question order bias often by thoughtfully considering the questions and what order I am asking the questions. You can also try randomizing the order of the questions although this may not make your user experience favorable if there are question category groups.
Social desirability bias:
Social desirability bias is a type of response bias in a quiz or survey that influences a participant to deny undesirable traits, and ascribe to themselves traits that are socially desirable.
Most of us want to be accepted and loved. We also think about how the data from our quizzes will be shared and in what way they will be shared.
An example of social desirability bias may be the following:
A company would like to know how good managers are to his/her team, so the company sends out a quiz to ask questions of the team members to rank the manager on a list of questions.
The team members may be concerned about the confidentiality of the survey data if the team members have negative things to say about his/her manager, and thus may give inaccurate answers to protect themselves against perceived retribution.
How many questions do you ask?
Have you ever taken a quiz so painstakingly long that you cancel a quarter or half way through? I have (many times)!
The problem is often that the quiz builder does not realize something about the user. The question of, “How many questions do you ask?” is a tricky question because the incentives of between the quiz taker and the quiz builder are often misaligned. Why do I say that the incentives are misaligned?
The quiz taker often wants as few questions as possible, so the more questions you introduce the more friction that you are causing to the user.
The quiz builder often wants as many questions as possible because they want to understand the user deeply.
So, how many questions do you ask? My strategy starting by going back to the “Acknowledge” question where I acknowledge who my customer is and what the paint point I am trying to solve for my customer. Once I know the problem, I hypothesize all of the questions that are needed to know how to solve that pain point for my customer. At the beginning, I brainstorm all of the questions that I could use to solve that customer’s pain point. After I come up with a list of questions, I receive input to determine which ones we can eliminate to solve the customer’s pain point. Do I need the user’s last name? Do I need both the phone number and the email? Why do I need both?
Also, I spend much of my time trying to think of ways to determine what I want to be answered without asking a question. An example of a question I ask myself, “I may know the IP address of the user, so do I really need to ask what state the user is in?”
By going through the process outlined above, I am confident that you will have a much better customer engagement with your quiz and much happier quiz takers.
Your goal as a quiz builder is to understand your user more to solve a customer’s pain point. It’s imperative that your goal is to build an objective data set by asking the right questions. To accomplish this goal, you must remove bias and ask the right questions.