Technology & Your Ministry Blog | Concordia Technology Solutions

How to Avoid Survey Bias

Written by Ann Ciaccio | Dec 27, 2016 3:00:00 PM

You’ve just hosted a webinar and want to know if you should host additional ones. The fund-raising event just concluded and you would like to know what your sponsors’ impressions were. Your congregation wants to move forward with an expansion of their building. A family ministry wants to determine key aspects of the relationship between parents, schools, and students.

These are just a few of the survey topics you may conduct for your church or school.

Surveys are intended to provide critical information for decision-making purposes. They uncover answers in an “anonymous” environment where people feel free to speak. Survey outcomes can evoke further discussions and discover trends—but only if the survey is written and executed properly and sticks to its goal.

What Is the Goal?

So, how do you write a survey in order to get the actual truth of the matter and not just the answers you want? Start by defining the goal of the survey.

Set a clear goal on what you want to achieve, and don’t stray from it. As an example, I’m going to use the survey our office sends out following the general pastoral conference held every year in our district.

The overall goal of the survey is to see if the conference serves the needs of our pastors in the district. It is not to see how good the conference was or to identify the “best part” of the conference.

Only ask questions that pertain to your goal or an objective that will help you achieve your goal. No matter how nice it might be to know other information, don’t ask questions that do not help you achieve your goal.

Who Is the Target Group?

Next, identify who your target group is. For our pastoral conference, the survey must be sent to all who are invited, not just those who came. The survey must identify why people did not attend. If it does not, the survey would be biased toward those who attended and we will miss the opportunity to serve those who did not attend.

Generally speaking, a survey should be sent to all stakeholders, anyone who has the possibility of being impacted by the results of the survey. Leaving out a group that may be affected will lead to incomplete results.

Making Certain Survey Questions Are Not Biased

Keep the tone of your survey balanced and evenhanded. Focus on creating clear questions and having an understandable, appropriate, and complete set of answer choices. 

We start our conference survey with the main question of attendance: Did you attend? For people who did not attend, we give a list of options and ask them to select the primary reason they did not attend. Likewise, for those who did attend, we ask the importance of factors in considering their attendance. This approach looks at attendance factors in similar ways from both the attendees and nonattendees in order to get a fair representation of both sides.

Don’t Lead Respondents

Biased questions lead to answers that support the researcher’s intent more than that of the respondent. Write your questions in a way that does not “lead” to a response.

For example, when rating the speakers at our conference, we never say how famous or well-known or well-liked they are. Rather, we simply ask, “How would you rate Tuesday’s conference program speaker, Rev. Dr. So and So?” We also add ratings on specific areas, such as interesting content, usefulness, and relevancy.

Have an Even Scale Rating

Unbalanced scales may be appropriate for some situations and promote bias in others. Our survey asks respondents to rate certain aspects of the conference, such as the speakers, accommodations, and registration process. In most cases, you should have a scale from Excellent to Very Poor. You should also include an opportunity for respondents to select “not applicable,” as they may have not experienced the particular aspect you are rating. You don’t have to have a large scale. Five points can provide a good picture.

Scale points should represent equidistant points on a scale. That is, they should have the same equal conceptual distance from one point to the next.

Cover All Possible Answer Choices

This can certainly be a two-edged sword. Too many choices will confuse your respondents, but not enough will not provide you with accurate information. Be sure to include choices you may not want to hear answers to.

And allow for an “Other” category for those who may have a reason to use it. Review the “Other” text your respondents have provided, and add the most frequently mentioned new options to the list for next time.

Avoid Loaded or Leading Words

Could, should, and might all sound about the same, but they may produce a difference in agreement to a question. Think about the words’ implications in this sentence: “Speaking louder leads to better participation.” Strong words such as prohibit may represent control or action and influence your results.

Consistency

If you survey a specific issue repetitively, it is important to use the same questions over time to compare those results. You may see trends occurring, and this will give you an opportunity to address them. 

Great questions and great answer choices lead to great research success. Define the goal of your survey, send it to all stakeholders, and keep your questions focused on the goal. Be open to any results! It will help better serve those intended!

 If you liked this post, subscribe to our blog and receive notification about new posts!