Survey Validation and Ensuring Congruity
Survey Question Development: How to Ensure Congruity
You might also like…
Analytics fit within descriptive, diagnostic, predictive, and prescriptive types and while each has its unique purposes, they add the most value when they all work in...
There are times when DIY research is a good solution and times when you need a managed research solution. So how do you decide...To DIY or Not to DIY: Is That...
As global competition becomes a growing concern for organizations across various industries, they increase their pursuit of excellence even more shrewdly. Brands are realigning their strategic goals and target across marketing, sales, and finance departments.
One way organizations are facilitating these goals is through extensive market research looking into customer experience improvements.
To make their research worthwhile, brands conduct market research to understand better their audience’s psychological attributes, buyer behaviors, and customer loyalty.
Survey-based instruments are used to measure and study different customer loyalty variables and metrics. This allows companies to illustrate trends and patterns that identify what’s working and what’s not.
For example, you can use NPS surveys to find what characterizes your promoters, passives, and detractors by assessing their willingness to recommend your product to their friends and family, what they think you could do to improve their experience.
However, when survey questionnaires are concerned, the instruments used must be valid. These allow you to collect accurate data and ensure the reliability of results obtained at the end of the research.
Often advisory pieces discussing and proposing survey development strategies mention validation as a running comment. They neglect to explain the methods used for validating a survey. This step is important as it allows the researcher to find whether the questionnaire measures what it was intended to measure.
Today, we’ll take a look at how you can build reliability and validity in your questionnaire and make sure they produce reliable and valid data and results.
Survey Question Development
Some of the traditional methods used to validate a survey include examining reliability coefficients, component analyses (factor, principal, or both), and correlation assessments between scores and/ or variables.
Before we take a further looking into survey validation, let’s take a look into survey reliability.
Survey reliability refers to the consistency of repeatability of the questions or other measurements. There are multiple ways of assessing and ensuring measure reliability.
One of the most commonly used is test-retest reliability, which assesses whether the questionnaire produces the same results when conducted multiple times under the same conditions (sample and setting).
The second method includes testing reliability within the measurement scale. This tests where every question or element meant to measure one variable indeed measure it.
Both tests are meant to determine whether there are consistency and repeatability in the survey. This is especially important for studies and research being conducted on an ongoing basis to detect any change.
With that said, let’s refocus on survey validity.
While reliability ensures repeatability, survey validity assesses the questions’ for dependability. There are multiple types of validity you can test. This includes:
- Face validity – This measures whether the questions measure the construct at hand. In other words, whether there is common sense in the questionnaire. This method relies on the researchers’ understanding of whether they know how people respond to survey questions and identify design flaws.
- Content validity – This measures whether all aspects of the variable being studied are covered in the design. Clear definitions are key here.
- Criterion or predictive validity – This measures whether the scores used in the questionnaire design can predict a criterion.
- Concurrent validity – This measures whether the questionnaire results align with the results of the measures being used in the study design.
To give you a more thorough understanding of how you can ensure validity in your questionnaire, follow the steps detailed below:
Establishing Face Validity
As mentioned in the previous section, face validity determines whether the questions capture the topics you’re researching. This step involves having the survey questions tested by two different parties.
The first group comprises people that are already familiar with the topic. The second party that reviews the questions includes subject matter experts that understand question construction.
They can identify whether there are any biases or errors, such as double-barreled or leading questions.
Run an Initial Test
A pilot test with your intended sample will iron out any issues within the survey and give you an idea of how effectively it measures the variables in your research model.
Choose a subset of the sample you’ve chosen and run the rest. The sample size you selected as participants for the pilot test can vary. However, 10% of your overall sample size is a solid place to start. While a higher number of participants is better, smaller samples can also work effectively in weeding out irrelevant questions.
Collect and Clean Data
Once you’ve administered the survey to your respondents, start recording their answers in a spreadsheet. The next step calls for you to clean the data to eliminate weal or erroneous answers.
This includes reviewing the values while reverse coding negatively phrased questions. This is to see whether the respondents’ answers to these questions match with similar questions you’ve phrased positively. You may want to remove the respondent from the survey if the answers don’t match.
In the same step, make sure you check the minimum and maximum values for your dataset. You can use a five-point scale to achieve this. If you see a number above the scale, it identifies an error. This might be an error in data entry or in the answer to begin with.
Principal Components Analysis (PCA)
PCA tests allow you to assess whether there are any underlying components in your survey questions. These components are there known as factor loadings. Questions assessing these components should load into the same loadings.
The scale testing the analyses is between -1.0 and 1.0. If the value for factor loadings is 0.6 or above, it’s a good sign. If a question doesn’t have factors, it’s better to remove or analyze them separately.
PCAs can be complicated to assess on your own. Research experts can help you out.
Internal consistency essentially checks whether there is a correlation between questions with the same factor loadings. It measures reliability and consistency. The standard test used to carry out this step is the Cronbach’s Alpha or CA.
N = # of items | c = average covariance | v = average variance
Look for values between 0.6 or higher as they indicate internal consistency. If your value is lower than 0.6, you can try deleting a question (one by one) to see if it improves the score
Revising the Survey
Once you’ve analyzed all questions in your survey design, revise it based on the results you generate during the validation process. Delete questions that don’t load onto a factor or lower the Cronbach’s Alpha to ensure that your measures are valid and dependable.
While experts can help you figure out the nuances of survey validity, there are programs and services available with pre-loaded algorithms that make this process easier for you. Utilize them to fortify the usefulness of your survey and guarantee that the results you obtain will be valid.
This step is critical for marketers and business owners engaging in market research as it aids in decision-making. The eventual result of these decisions has an impact on customer experience and loyalty. Starting on the right foot will help you assess how successfully you improve your business’s performance.
OvationMR is a global provider of first-party data for those seeking solutions that require information for informed business decisions.
OvationMR is a leader in delivering insights and reliable results across a variety of industry sectors around the globe consistently for market research professionals and management consultants.
Need help with your project?
We are ready to offer you: