Where does this survey question go?
If you find yourself searching for the best way to ask and organize your survey questions for a market research study, company feedback, or academic paper you are not alone. Writing survey questions that accurately represent public opinion is the number one concern for people who are new to designing surveys and professional researchers alike.
Asking questions in the right way and the correct order is necessary to reduce error which could impact your data and ultimately the findings of your study. Managing the following types of errors goes hand-in-hand with sampling design in order to increase the precision of your survey results.
This article will provide survey questions examples to help you reduce or eliminate:
- Respondent bias
- Data errors due to question ambiguity
- Missing data due to logic errors
To get started on the right foot it’s important to develop a plan for your online survey which includes a questionnaire structure that works universally. This system should be flexible enough to adapt to the specifics needs of your particular research study. An online survey questionnaire can be thought of as a hierarchy that includes: Sections and Questions in Sections.
Online Survey Sections and Flow Diagram
This is where respondents…
- enter before they are qualified
- are asked qualifying questions
- presented these in a blind list
- if pre-qualified are re-qualified
- not qualifying are terminated
- in a filled quota-cell are termed
Collect primary findings about…
- awareness and usage
- product or service ratings
- concept testing
- polling and public opinion
- customer satisfaction
- employee feedback
Collect classification data like…
- personal (age, gender)
- household (children, income)
- economic (employment, class)
- geographic regions
- other (education, relationship)
- B2B (industry, revenue)
How do you create effective survey questions?
The good news is we have tools to help us craft more structured while still engaging online surveys. Tools that have been designed to reduce the time it takes to build an online survey and also improve the data quality by enabling data validation and edits in real-time.
These capabilities along with respondent engagement features like question gamification, image response lists, and video help improve the overall experience and response rate of surveys.
What are examples of survey questions you can use to create effective online surveys?
With so many different survey question options, how do you choose the right ones for your needs? Just like craftsmen have different classes of tools; ones that measure, others that cut things, and so on – so do we as survey designers. Broadly speaking survey questions used individually or in combination with others help you to:
- filter respondents answering all or part of a question or survey section
- solicit categorical responses
- measure changes in perception/attitudes/opinions
- sort items into groups
- build lists
- ask probing questions and solicit a verbatim response
These are just a few examples of the main things you will be doing when creating your survey. Survey questions are the tools we use to build online surveys and generally speaking, there are the questions you answer from a pre-defined list known as a Closed-end Question and those that are unstructured responses known as Open-End Questions.
Single-choice response question type
A single-choice response question allows the flexibility to present a predefined list of items to a respondent while allowing for only one item to be selected.
This is why it’s the perfect choice question for most classification questions that you will probably design like this demographics question about employment:
A few features that help reduce bias and improve data quality to consider when using this question type are:
- The ability to include a special item called the “Other” category that can be set to allow a respondent to enter their own response.
- The ability to randomize or rotate the presentation of the response lists to reduce order bias on preference and awareness questions
There are many instances where a single choice question can be the perfect fit for your online survey. Here are some ideas which you can try out yourself:
|Binary choice:||Paired preference:||Best selection:|
|Yes | No||Pepsi or Coke||Voting|
|True | False||Mac or PC||Classifying|
|Opinion | Fact||B.I.G. or Tupac||Testing|
Whether you’re picking your favored candidate or favorite color the single-choice response question is a primary tool you will use often.
Dropdown list question type
A dropdown list is the simplest method when featuring a long response list. There are many types of extensive lists where you will want to offer a respondent an alternative to a screen full of answer choices. For example, when dealing with a classic long list scenario like selecting the country of birth is to use a dropdown question like this one:
There are alternative methods for solving the long list problem which can be equally as effective. For example, there are occasions when you want to allow a respondent to just start typing in their answer and then have a search function auto-populate their likely narrowed down choices or exact choice. This is easily done with simple code and a lookup table that your survey programming partner can help you with.
It is important to consider the use of the dropdown on a case by case basis. Not all questions are meant to be slowly revealed to the participant. There is clearly a case to be made in circumstances like choosing preferences for people to see all the choices at one time before choosing.
Multi-choice response question type
A multi-choice response question like the single-choice response allows the flexibility to present a predefined list of items. This question type allows a respondent multiple selection choices including one or more “Other” response options.
This is why its the perfect question choice for most classification questions that you will probably design like this usage and behavior question about business travel:
Like single-choice response questions, you can randomize and rotate lists. Since the lists are a multi-choice response, you have the option of limiting the number of responses given. For example, you may present a list of 10 types of snack foods and ask people to only choose their top 3 snacks.
You can also make items on a list mutually exclusive meaning if they checked that item, then the other items would not apply. For example, a pet store might have a survey asking people what fish would they like to see in a fishbowl or tank? A goldfish, an angelfish, a betta fish, and a fancy guppy. In this online survey, the pet store would make the response betta fish mutually exclusive because a betta fish should be in a tank or bowl by itself.
The use of single-choice response questions combined with a multi-choice response question is a powerful often used tool for building brand awareness and usage and attitude insights. They are the workhorses of online survey tools and together can be the perfect fit for your next project.
Multi-choice card sort question type
A multi-choice card sort is our recommendation for a refreshing alternative to the matrix question or grid question. This is a great tool because it’s mobile-friendly and takes up less real-estate on the screen than matrix or grid question formats.
Here is an example of a user categorizing four venues and indicating what technology they used in each venue:
There is also a single-choice card sort option for this question type that makes it the ideal alternative to using matrix questions for rating multiple brands or product features using Likert scales. Using the card sort respondents may rapidly loop through a series of brand attributes and rate them. Unlike a matrix question, users can’t rapidly straight-line their answers by clicking boxes across or down the screen without a thought.
Likert-scale slider question type
If you have ever been asked if you were neither satisfied nor dissatisfied with the quality of your cellular service, chances are you were in the middle of doing a survey and answering a classic Likert scale type question. Likert questions are great for measuring people’s feelings: extremely satisfied – not at all satisfied, or level of agreement: strongly agree – strongly disagree.
Here is an example of a Likert scale question being used to measure satisfaction with a recent hotel stay during a business trip. The question is being collected using a slider tool which gives the user the feeling of increasing and decreasing while still allowing five data point options:
Likert questions are also suitable for measuring people’s propensity: extremely likely – not at all likely, to purchase, vote, volunteer, respond or change their behavior in some way and perhaps give you an indication of how important it is:
- very important
- moderately important
- slightly important
- not important
So whether you use a Likert scale question in a slider a single-choice response question or card sort, it’s a great tool for measuring. From an analysis point of view, it has a midpoint with equal measures above and below it. Making it the perfect choice for statistical analysis and drawing inferences from the data.
Rank sort question type
Another way of soliciting preference data from users is to leverage ranking questions. Ranking questions go beyond asking which brands or products they like collectively. Ranking forces participants to have more time to think about preference and order; Which for some users in some situations can increase overall survey duration and may create conflict or annoyance when overused.
Here is a ranking sort version of a question asking business travelers what their favorite cities for business events are in order of preference:
Open-end text question type
Sometimes you just have to ask and that is what the open-end response text question is designed for. Whether it is a short form or long-form type of question and response, respondents have the opportunity to tell it – say it – type it – text it and even video it… like it is.
Here is an example of a long-form open-end text question asking people to describe their favorite summer vacation destination experience:
This open-end text question can easily add multi-media options such as video uploads or audio recording.
Analysis of open-end data varies widely among different users. The traditional market researcher typically will conduct a process known as coding open-ends. This process basically takes the unstructured data and organizes it through a numeric classification process assigning respondent’s statements, brand and product mentions, and sentiments into groups so that the coded data can be tabulated with the other survey data.
Other users leverage the verbatim text alone as qualitative insight data to act on in concert with the survey data. While some process the unstructured data into word clouds and other data post processors.
Open-end numeric question type
When only a number will do enter the very versatile open-end numeric response. This question allows you to set the type, format, and range of any number. Which is great for collecting a wide variety of data from pricing to temperature and number of days to average monthly rainfall to two decimal places.
In this example we are asking people how many days they worked from home last week:
The max range is 0 to 7 however, we could easily have a multi-choice response question before this one asking which days of the week do you usually work. If the respondent chooses M-F and Sat we can count 6 items selected for that question and set the numeric range for the question above to 6 instead of 7 for this respondent.
To take the idea a step further if the respondent answered M-W-F and Sat to the setup question: which days of the week do you usually work – you can follow this up with a condition two questions:
- How many days did you work from home during the regular business hours (limiting the max range to 3 because they only selected M-W-F)?
- How many days did you work from home during the weekend (limiting the max range to 1 because they selected only Sat previously)?
There are infinite possibilities for combining different question types together to improve the survey experience, create custom formulas based on user inputs, and increase the value and the quality of your data collection effort.
Tips for writing survey questions that get you accurate results
Now that we have reviewed some great options for you online survey questions we think it’s important to point out the number one factor you can control to improve your data quality is to control who is coming into your survey first and foremost and then control the experience they have while doing the survey.
That means… make the survey experience great…
Ensure you’re reaching the right respondents
Writing the survey may seem like a challenge enough before having to worry about where respondents are going to come from and how to ensure your getting the right people, but it doesn’t have to be. Here are some simple things to remember to keep you on track:
- Identify the target population during the sample design process
- Work with sample provider to ensure precise sample panel targeting
- Slow start your project and check your data before full-launching.
- For general population studies, your demographics questions will indicate representation
- When screening for low-incidence qualifiers, prioritize sampling for male quota cells before female cells
- B2B surveys to ensure they are reaching the right target should rescreen the respondent
- Ensure survey end links to report complete, terminate, and over-quota are working so sample providers can properly send participants
Let the survey tools do the heavy lifting for respondents and for you
While there are countless questions you can think to ask people and as many ways of asking them it’s often easy to overlook the fact that your survey tool does more than just simply present a question followed by a qualified response list. Much more.
In this article, we have already shown you ways you can combine questions together to smarter about asking questions more accurately later in the survey. There are many more opportunities to expand on this and improve the survey experience by helping the respondent from having to work so hard, Here are some examples:
- Auto-calculating sums and percentages
- Ratings based on brands/products either aware/used/purchased
- Forms and calculation fields to answer complex questions simple, fast and accurate
- Question gamification like card sorts to increase engagement
- Heatmaps to analyze where people click on pages or images.
- Text highlighting to help with message testing and concept testing.
With some new suggestions to try out, keep in mind it’s always important to review your data and compare performance with past results. If you are doing any kind of brand tracking or benchmarking there is a case to be made to look at things like missing data for opportunities to improve study design.
The opportunities are endless to be creative and expand upon what others have done! Good luck and good programming!
Jenn Whaley leads the Insights Solutions and Customer Success Teams for OvationMR and posts frequently on The Standard Ovation and other Industry Blogs. OvationMR is a global provider of first-party data for those seeking solutions that require information for informed business decisions. OvationMR is a leader in delivering insights and reliable results across a variety of industry sectors around the globe consistently for market research professionals and management consultants. Visit: https://www.ovationmr.com.
Need help with your project?