Market Research » Survey Validation and Ensuring Congruity

Survey Validation and Ensuring Congruity

Survey Question Development: How to Ensure Congruity

You might also like…

 As global competition becomes a growing concern for organizations across various industries, they increase their pursuit of excellence even more shrewdly. Brands are realigning their strategic goals and target across marketing, sales, and finance departments.

One way organizations are facilitating these goals is through extensive market research looking into customer experience improvements.

To make their research worthwhile, brands conduct market research to understand better their audience’s psychological attributes, buyer behaviors, and customer loyalty.

Survey-based instruments are used to measure and study different customer loyalty variables and metrics. This allows companies to illustrate trends and patterns that identify what’s working and what’s not.

For example, you can use NPS surveys to find what characterizes your promoters, passives, and detractors by assessing their willingness to recommend your product to their friends and family, what they think you could do to improve their experience.

However, when survey questionnaires are concerned, the instruments used must be valid. These allow you to collect accurate data and ensure the reliability of results obtained at the end of the research.

Often advisory pieces discussing and proposing survey development strategies mention validation as a running comment. They neglect to explain the methods used for validating a survey. This step is important as it allows the researcher to find whether the questionnaire measures what it was intended to measure.

Today, we’ll take a look at how you can build reliability and validity in your questionnaire and make sure they produce reliable and valid data and results.




Survey Question Development


Some of the traditional methods used to validate a survey include examining reliability coefficients, component analyses (factor, principal, or both), and correlation assessments between scores and/ or variables.

Before we take a further looking into survey validation, let’s take a look into survey reliability.

Creating and image for an online concept test market survey

Survey Reliability


Survey reliability refers to the consistency of repeatability of the questions or other measurements. There are multiple ways of assessing and ensuring measure reliability.

One of the most commonly used is test-retest reliability, which assesses whether the questionnaire produces the same results when conducted multiple times under the same conditions (sample and setting).

The second method includes testing reliability within the measurement scale. This tests where every question or element meant to measure one variable indeed measure it.

Both tests are meant to determine whether there are consistency and repeatability in the survey. This is especially important for studies and research being conducted on an ongoing basis to detect any change.

With that said, let’s refocus on survey validity.



Survey Validity


While reliability ensures repeatability, survey validity assesses the questions’ for dependability. There are multiple types of validity you can test. This includes:

  • Face validity – This measures whether the questions measure the construct at hand. In other words, whether there is common sense in the questionnaire. This method relies on the researchers’ understanding of whether they know how people respond to survey questions and identify design flaws.
  • Content validity – This measures whether all aspects of the variable being studied are covered in the design. Clear definitions are key here.
  • Criterion or predictive validity – This measures whether the scores used in the questionnaire design can predict a criterion.
  • Concurrent validity – This measures whether the questionnaire results align with the results of the measures being used in the study design.

To give you a more thorough understanding of how you can ensure validity in your questionnaire, follow the steps detailed below:



Establishing Face Validity


As mentioned in the previous section, face validity determines whether the questions capture the topics you’re researching. This step involves having the survey questions tested by two different parties.

The first group comprises people that are already familiar with the topic. The second party that reviews the questions includes subject matter experts that understand question construction.

They can identify whether there are any biases or errors, such as double-barreled or leading questions.



Run an Initial Test


A pilot test with your intended sample will iron out any issues within the survey and give you an idea of how effectively it measures the variables in your research model.

Choose a subset of the sample you’ve chosen and run the rest. The sample size you selected as participants for the pilot test can vary. However, 10% of your overall sample size is a solid place to start. While a higher number of participants is better, smaller samples can also work effectively in weeding out irrelevant questions.



Collect and Clean Data


Once you’ve administered the survey to your respondents, start recording their answers in a spreadsheet. The next step calls for you to clean the data to eliminate weal or erroneous answers.

This includes reviewing the values while reverse coding negatively phrased questions. This is to see whether the respondents’ answers to these questions match with similar questions you’ve phrased positively. You may want to remove the respondent from the survey if the answers don’t match.

In the same step, make sure you check the minimum and maximum values for your dataset. You can use a five-point scale to achieve this. If you see a number above the scale, it identifies an error. This might be an error in data entry or in the answer to begin with.



Principal Components Analysis (PCA)


PCA tests allow you to assess whether there are any underlying components in your survey questions. These components are there known as factor loadings. Questions assessing these components should load into the same loadings.

The scale testing the analyses is between -1.0 and 1.0. If the value for factor loadings is 0.6 or above, it’s a good sign. If a question doesn’t have factors, it’s better to remove or analyze them separately.

PCAs can be complicated to assess on your own. Research experts can help you out.



Internal Consistency


Internal consistency essentially checks whether there is a correlation between questions with the same factor loadings. It measures reliability and consistency. The standard test used to carry out this step is the Cronbach’s Alpha or CA.

Cronbachs Alpha Formula for survey validity

N = # of items | c = average covariance | v = average variance


Look for values between 0.6 or higher as they indicate internal consistency. If your value is lower than 0.6, you can try deleting a question (one by one) to see if it improves the score



Revising the Survey


Once you’ve analyzed all questions in your survey design, revise it based on the results you generate during the validation process. Delete questions that don’t load onto a factor or lower the Cronbach’s Alpha to ensure that your measures are valid and dependable.

While experts can help you figure out the nuances of survey validity, there are programs and services available with pre-loaded algorithms that make this process easier for you. Utilize them to fortify the usefulness of your survey and guarantee that the results you obtain will be valid.

This step is critical for marketers and business owners engaging in market research as it aids in decision-making. The eventual result of these decisions has an impact on customer experience and loyalty. Starting on the right foot will help you assess how successfully you improve your business’s performance.





Jim Whaley

Jim Whaley



Jim Whaley is a business leader, market research expert, and writer. He posts frequently on The Standard Ovation and other industry blogs.

OvationMR is a global provider of first-party data for those seeking solutions that require information for informed business decisions.

OvationMR is a leader in delivering insights and reliable results across a variety of industry sectors around the globe consistently for market research professionals and management consultants.


OvationMR Survey Validation Contact Form

Need help with your project?

We are ready to offer you:


Our latest Panel Book


ESOMAR28 Response


A project estimate/proposal


39 Broadway, Suite 2010, New York, NY 10006 USA