Survey Best Practices

Ten Steps to Successful Surveys

• 1. Ask Yourself: Do You Really Need to Conduct a Survey?

  There are many good reasons to gather information through surveys; however, before embarking on a survey journey, it is important to ask yourself if conducting a survey is the best approach to obtaining the information you need.

  A survey is not a good idea when	Alternative ways to collect information include
  information can be obtained elsewhere; you are unable to act on the information collected; you cannot articulate a clear objective; you do not know how to analyze the data; you cannot guarantee that a representative sample of participants will respond; you need to ask a lot of open-ended questions; you cannot tell participants how your survey will benefit them.	focus groups or interviews; institutional data; existing survey data; observational data.

  When a Survey is the Right Method

• 2. Create a Reasonable Timeline

  While it may seem simple enough, creating a quality survey takes time. If you do decide to conduct a survey, you are required to submit a Survey Approval Request Form to the Office of Institutional Research & Assessment.

  Additionally, you will want to pilot test your survey. This allows you to identify problematic instructions or questions. Questions that are misinterpreted will not be useful to you, so make sure to get the most out of every question.

  Consider the following timeline:

  1. Review survey instrument with stakeholders and Office of IRA.
  2. Create a communication and publicity plan.
  3. Survey instrument and plan are approved and finalized.
  4. Build survey in survey platform.
  5. Test survey through survey platform.
  6. Execute communication and publicity plan.
  7. Survey opens – Invitation sent.
  8. Reminder email 1.
  9. Reminder email 2.
  10. Final reminder or Survey deadline extended email.
  11. Survey Closes.
  12. Analyze Results.
  13. Report Results.
  14. Actions.
  15. Archive or dispose of data.
• 3. Define Your Survey Goals

  Define a clear, attainable purpose for your survey supported by three to five goals. The purpose is a broad statement of the primary survey aim or outcome. The goals are specific and measurable ways to achieve that purpose.

  State your purpose using an action verb, such as describe, explain, explore, identify, investigate, gauge, measure, assess, or test.

  • Identify difficulties in obtaining professional training in order to make skill development more easily attainable.
  • Gauge training effectiveness to make improvements to next year’s training.

  Goals help you determine your target audience, how best to distribute the survey, and how to design questions that are relevant and lead to actionable outcomes.

  • Goal 1: To assess various factors that lead users to disengage such as time of year, length of training, method of delivery, etc.
  • Goal 2: To determine what devices users employ to take part in a training module.

  The purpose and goals will help you provide context and set expectations for your survey audience, which is vital to engaging them with the survey and obtaining valid results.

  Learn More About Defining Goals

• 4. Write Your Survey Questions

  Writing good survey questions and organizing them sensibly makes it easier for respondents to answer truthfully and complete the survey. Your aim is to make completing the survey as easy as possible for the respondent.

  To gather opinions and behaviors, consider what type of rating scale will produce the best, analyzable results from your multiple choice questions.

  Important considerations for writing questions include the following:

  • All questions should support the survey goals and purpose.
  • Keep the survey short.
  • Multiple-choice answers should be clear, distinguishable and balanced.
  • Consider mixing different question types.
  • Place personal or demographic questions at the end of the survey.
  • Use only a few open-ended questions, and include them toward the end.
  • Don’t ask leading questions. Ensure questions are neutral, not biased.
  • Don’t ask for two things at once (avoid the “double-barreled” question).
  • Use skip logic as needed.
• 5. Select the Sample

  The Who

  Selecting the target audience is vital to obtaining the actionable insights you seek. The target audience is based on a set of demographics, behaviors, or characteristics.

  Suppose you are surveying students. Do you want to include undergraduate/graduate, traditional/adult, part-time/full-time, residential/commuter, in-person/online? Students who studied abroad, had an internship, were in a club? This is just one example of “audience segmentation.” Targeting the correct, specific population is crucial to the success of your survey project.

  How Many

  The sample population must be of sufficient size to allow you to generalize your findings in a way that accounts for diversity and heterogeneity within the population. The sample of respondents needs to be representative of the population’s important characteristics to allow for valid inferences from the data. (A representative sample “accurately represents the characteristics of a larger population.”)

  Calculate Your Sample Size with this Tool

• 6. Select a Survey Platform

  There are multiple ways to administer a survey, including one-on-one interviews, mail, and online. Chances are, you will be using an online platform. Considerations include user privacy and data security.

  Self-administered formats include Google Forms as well as freeware such as SurveyMonkey. For a high-stakes survey, work with the Office of Institutional Research & Assessment. We will build the survey and administer it for you through Champlain’s licensed Qualtrics account.

  Ways to Collect Data Via an Online Platform

• 7. Consider Incentives & Publicity

  Consider how publicity and incentives can bolster your survey’s response rate and the likelihood of obtaining results from a representative group. Whether you are emailing survey links, conducting a survey in class, or posting a QR code to provide survey access, you will need to promote your survey in order to get a good response rate and a representative sample.

  In promoting your survey, it is helpful to emphasize the survey’s value to the respondent or to something they care about. You can advertise, for example, on social media, The View, digital signage, the Stall Street Journal, the Academic Brief, the People Center Brief, The Crossover, posters, flyers, or verbal announcements (at Faculty Senate or Staff Council). It may be appropriate (for a high-stakes, institutional survey) to send a pre-survey announcement (from a high-placed administrator) by mail or email.

  Incentives can improve response rates, but also pose potential negative consequences. Incentives should be large enough to entice the respondent, but not so large as to overshadow the importance of the survey itself, or worse, constitute “undue influence.” A small incentive for each respondent who completes the survey (a $5 gift card to Zime) or a chance to win a larger prize (a $30 Amazon gift card, an Apple watch) through a lottery constitute acceptable incentives.

  If incentives are used, promotional materials as well as the survey invitation and reminder communications must describe the prizes, their value, and an estimate of the odds of winning.

• 8. Data Security

  Surveys ask respondents for their opinions and personal information. As the initiator and owner of a survey project, you are responsible for making sure your project respects privacy and data security.

  If you collect personal data, tell survey takers how you plan to use their data and how it will be stored, protected, and potentially destroyed. Sharing your privacy practices with survey takers may help ease their privacy concerns and increase your response rate. Describe your privacy practices as well as the anonymity or confidentiality of the survey results in the survey introduction and in the email inviting people to take your survey.

  Understand the Security Practices of Your Platform

  How data is stored, used and protected varies from platform to platform.

  If the survey is built on the Qualtrics platform, it is secure; “Qualtrics complies with applicable data privacy laws in its role as a data controller of personal data it collects,” and is GDPR compliant.

  However, confidentiality and privacy of data collected through Google Forms depends largely on the user’s settings, making user vigilance an essential aspect of data security. Google Forms should not be used for collecting sensitive information unless additional security measures are in place. 

  Learn About Champlain College Data Security Policies

• 9. Distribute Your Survey

  Once you have chosen the platform, you can choose from different options for distributing the survey, for example:

  1. Create a custom URL for your survey, then get responses by emailing the link through your own email.
  2. Upload an email list and build custom email invitations in the survey platform and send your surveys automatically by email through the platform. This is the only method that will allow you to track whether respondents have taken the survey, and send reminders to those who haven’t responded.
  3. Embed a survey link on a website.
  4. Publicly post a survey link on social media.
  5. Print a QR code on posters and flyers.
  6. Share a survey link with a group of people in a meeting or class.
• 10. Analyze Your Results

  If you are unfamiliar with analyzing quantitative and qualitative data, ask for help from the Office of Institutional Research & Assessment.

  Measurement Types

  There are four different measurement scales that shape data analysis:

  • Nominal (or Categorical): “Named” data categorized (with no specific order) into mutually exclusive groups (i.e. class level, home state, location, gender, race/ethnicity, major, job type)
  • Ordinal: Variables that occur in a specific order (i.e. a rating or ranking that measures frequency, satisfaction, degree, grade)
  • Interval: Variables that occur in a specific order with equal, quantifiable intervals between values with no true zero (i.e. temperature, date, income, or time)
  • Interval: Variables that occur in a specific order with equal, quantifiable intervals between values with no true zero (i.e. temperature, date, income, or time)

  Learn About Question Scales

  Data Analysis

  Data analysis allows you to make inferences from the survey data that can lead to insights or actions. Analysis is done by conducting operations on the data. A good survey–which asks about measurable, observable characteristics, activities and behaviors–provides analyzable data. Use statistical analysis to determine if comparisons or trends have statistical significance rather than being due to chance.

  Most likely, you will conduct descriptive analysis, which summarizes and describes the important characteristics of the data set, mainly frequency, central tendency (averages, including mean, median and mode), and dispersion.

  Cross-tabulation allows you to break out the data according to sub-groups within the survey population for comparison. Also known as contingency tabulation, cross-tabulation is subject to tests of statistical significance including chi-square. Cross-tabs can be created in Excel using pivot tables.

  Whereas quantitative data answers the “what,” qualitative data enables you to dig into the “why” and “how.” A word cloud is a quick and easy approach that uses an image, illustrating the most frequently used terms. More rigorous qualitative analysis methods include:

  • Content analysis: systematically analyzing a text to identify certain features or patterns.
  • Thematic analysis: identify patterns and themes in data.
  • Narrative analysis: identify, analyze, and interpret the stories that customers or research participants tell.
  • Sentiment analysis: natural language processing that determines the emotional undertone behind a text through artificial intelligence.

  Reporting

  Once your data analysis is complete, share the story. A formal report should include:

  • the survey purpose and goals, including the impetus for and story behind the research question;
  • the survey method used, including target audience, data collection and analysis methods;
  • survey respondent descriptive statistics (representativeness);
  • survey results, including statistical findings and takeaways; and
  • a summary that includes important findings and next steps.

  Use visuals (charts, graphs and tables) to support text. You may wish to track how the findings for particular questions have changed over time (trend analysis). Take care to contextualize and not overinterpret data. Do not confuse correlation and causation.