Developers Corner: Survey Writing

Developers preparing feasibility studies and business plans often use a survey to gather quantitative and qualitative data. Surveys are helpful for other applications too. For example, a Center might want to understand client needs for strategic planning purposes. One of the most effective tools to do this is a survey. Taking the time to do it right will pay off with reliable responses.

Online surveys are being used with increasing frequency and commercial software such as Survey Monkey or Survey Gizmo are available. These make fielding a survey much easier with pre-formatted question types and example questions. Even with helpful software, designing a successful survey still requires thoughtful development. This Developer’s Corner focuses on tips for survey design.

A survey can fail to achieve a good response rate or produce worthless data for several reasons. The following are our top 10 hints for good surveys.


Make it short.

We all know this rule but more times than not when asked to review a survey, it is too long. Resist the urge to add too many questions. For an online survey, try for 10 questions. A survey that is too long will either discourage someone from starting or they will drop out part way through.

Look at each question with your final report in mind. Only include questions that directly shed light on what you want to learn. If the question seems like a minor contribution to the learning, leave it out.

Be sure to remove questions that require the respondent to have immediate access to specific information not readily available. Warn the respondent beforehand if they need specific information if the question is critical to your goals. Take out any questions that a respondent may not be willing to answer unless the information is absolutely crucial to what you need to know. For example, if knowing income is not critical, don’t ask about it.


Name it carefully.

This seems like a small hint, but the name of the survey should be catchy and inviting. What appears in the tagline or in the subject line of an email soliciting respondents and can be a “make it or break it” phrase for the person deciding whether to spend the time responding. Grab their interest quickly. If you are aiming at respondents from a specific geography, put that in the title. Thus Warren County Local Food Survey is better as title than Food Cooperative Survey.

---

Provide clear motivation
​
Provide clear motivation on the “why” of the survey. Use short sentences and concise language.

Bad: This survey delves into the need for a local food cooperative to advance the local food system. We are a group of concerned citizens who want to expand access to local foods through a cooperatively owned grocery store.

Better: We need your opinion on the local food system in Warren County. Your answers will help us understand the gaps in local food access. One of our goals is to open a cooperatively owned grocery store. We are trying to understand if this is a good thing to do.


Order the questions carefully.

Start with the interesting questions that will attract interest. Put the simple questions upfront. Put personal questions like age or income which are more likely to discourage a response toward the end. Also, put open-ended questions toward the end. Get as much information as you can with easy to answer questions and put those requiring the respondent to write something later in the survey. That way you obtain some info before the respondent decides the time is up and opts out.


Stick to simple wording

Your respondents will likely have a wide range of education and reading ability. Make your questions concise both for the respondent’s understanding and to keep the response rate up. A good question should be short and straightforward.

Bad: What is the frequency of your trips to your primary grocery store in an average week?
Better: How many times a week do you usually shop for groceries?


Also avoid concepts that require too much background information for the respondent to easily answer. In addition, avoid questions that require too much recall.

Bad: In the past two years, how many times did you and/or your family members travel outside your community to grocery shop?
Better: In the past month, how many times did you drive over 5 miles to grocery shop?


Watch out for double negatives. They make the question difficult to understand.

Bad: My local grocery store is not failing my needs.
Better: My local grocery store meets my needs.


Be specific in the wording of your question to focus the information gathered.

Bad: Do you like your current grocery store?
Better: What do you like about your current grocery store? (Then provide a list)


In addition, voicing questions in the third person can be less threatening. Compare these two questions:

How do your colleagues feel about the current management decisions?
How do you feel about the current management decisions?


Structure the question to be singular.

Be careful that your question only asks about one thing. Look at each question to be sure you have focused on only one concept.

Bad: Availability of local and organic produce determines where I shop for groceries.
Better: Availability of local produce determines where I shop for groceries.
Better: Organic produce availability determines where I shop for groceries.

Bad: How organized and interesting was the speaker?
Better: How organized was the speaker? (provide a ranking from which to choose)
Better: Was the speaker interesting?


Cover all the answers

The response alternatives should be exhaustive and mutually exclusive.

Bad: How many times do you shop for groceries in an average week?
a) 1 b) 2 c) 3 d) 4 e) 5 or more

Better: How many times do you shop for groceries in an average week?
a) 0 b) 1 c) 2 d) 3 e) 4 f) 5 or more


Keep your bias out

Writing survey questions that bias respondents or leads them toward one answer violates a survey’s objectivity and biases the answers you get. Keeping the tone of your survey balanced and even-handed will ensure that you get people’s “true” attitudes instead of what they think you want to hear. This will help you make the right decisions, and alert you when you have a problem.

Bad: How awesome do you think a food cooperative would be in Sommersville?
Better: Do you think Sommersville needs a new grocery store?

Bad: Would you shop at a food cooperative if it offers quality food at lower prices?
Better: What would a food cooperative need to offer for you to shop there? (provide list)


Limit open-ended questions.

Open-ended questions will often ask more from your respondent than she or he is willing to give. Plus they make analysis much more difficult. What do you do with 600 written answers? You will spend a lot of time sorting them into categories for a summary. With that said, giving the respondent who has something more to say a place to do that is important.


Do a Pretest.

First develop your survey, and then choose a few people to fill it out. This pretest will highlight questions that are problematic or incomplete. It will also tell you if you are getting the appropriate data to meet your goals. Many a survey has been saved from failure by a pretest.

And remember to thank your respondent for helping.

By: Cathy A. Smith, Ph.D.