NAIT 2014 PD day presentation: A step-by-step guide to developing better, more useful surveys

summary of pd presentation

Hi all,

Apologies for the long delay since our last post! We’ve been busy working on other projects for the institution including launching the Student Engagement survey and preparing the Entering student survey to be launched Fall 2014.

We also spent some time preparing a presentation for NAIT Professional Development days (June 4th and August 28th). Our presentation focused on key steps involved in developing a survey, including:

  • Set your objectives (What do you need to know?)
  • Sampling (Who knows what you don’t know?)
  • Choose a methodology (How are we going to learn what they know?)
  • Develop strong survey questions
  • Launch survey, monitor results
  • Clean data/explore findings
  • Interpret results (through the lens of your objective)

The five key takeaways from our session were:

(1)    The importance of defining your survey objectives. This is important because it prevents scope creep (avoiding “wouldn’t it be interesting to know X….” questions) and survey length. Objectives are also important as a focusing tool during the analysis phase.

(2)    Survey development is an iterative process. Despite the step-by-step format we delivered the presentation, developing a survey is an iterative process. Throughout the survey development process (and into the analysis phase), we consider how a decision at one step will impact other decisions we have made. For example, you may refine or change your objectives based on a discussion about sampling or methodology.

(3)    Question wording is important. Developing survey questions is as much as a science (avoid double barreled and leading questions) as an art form (know your audience, consider emotional triggers that may bias responses).It’s always a good idea to pilot your survey with a few members of your sample population to ensure your questions are clearly written and easy to understand.

 (4)    Be careful when interpreting results (and reading charts from other researchers or media publications!). Remember that correlation does not equal causation (just because two factors are correlated doesn’t mean that one factor is causing the other to occur – see example below).

correlation does not equal causation

Also, always consider whether the author is purposely presenting data in a way to make a point. See Media Matters terrific post covering “A History of Dishonest Fox [News] charts”  for examples of misleading charts.

 (5)    NAIT Institutional Research department is here to help as you develop surveys or interpret results!

We’ll be running another session on August 28th. We encourage all NAIT staff members to attend!

We put together a guide for best practices in survey design that may be helpful as you develop your own surveys. You can access it on our website here.


Survey best practices – (2) choose an audience and delivery method

studentsIn the last post, we discussed the importance of setting concrete objectives as a first step in developing a survey. Once you have clearly articulated your objective, the next step is to identify who you want to fill out the survey. After going through the process of developing your objective you should have some idea of what your sample population will look like.

In this step, you want to make explicit who will be included (and excluded) in your survey sample as well as decide whether information collected will be confidential (name included, but not shared with anyone) or anonymous (name not asked of respondents).

Your survey sample should reflect your survey objective. Who you decide to survey will change depending on what your research objectives are. For example, consider the required survey sample for the following objective:

To review student’s food preferences in the school cafeteria in order to better understand what students would like to eat while on campus so we can make informed decisions about food services at X institution going forward.

We obviously want to sample X institution’s students, but we need to be more specific:

  • Should we ask all enrolled students (on-campus, at satellite campuses, online learners)?
  • Should we ask only those students who actually eat on campus or all students to examine why these students choose not to eat on campus?
  • Should we ask both credit and non-credit students?

My point is not to be exhaustive with the above questions but rather to highlight the importance of considering who you want to included (and exclude) from your sample.

At this point you should also consider how much information about the respondents you want to collect and whether survey responses will be merely confidential or anonymous. Unless you plan to follow-up with respondents (and therefore need to know their name and contact details) or link their responses to another data set (for example, linking Student Engagement Survey responses to responses on an Entering Student Survey to track changes in views/attitudes over time) anonymous responses is likely sufficient.

Once you have decided on who your sample is going to be and whether their responses will be treated anonymously or confidentially, the next step is to decide on a data collection method. There are a number of options, including mail-outs, telephone, face-to-face interviews, and web-based (email) surveys. Each option has associated pros and cons such as cost, human resource (i.e. time) requirements, and response rate. The method you choose should align with the objective of the survey with consideration of the population being surveyed. Other realities such as cost and delivery deadlines may also influence this decision.

In our example, the cost and time commitment to complete a telephone, F2F, or mail-out survey would be far too great. Moreover, most students have access to email so our chosen data collection method is fairly clear – email invitations to participate in a short survey.

At this point, you should have:

  • Developed a clear and detailed survey objective
  • Developed a description of your sample, including who is included and who is NOT included
  • Made a decision whether your sample’s responses will be treated confidentially or anonymously
  • Chosen a data collection method based on your objectives and survey sample

For the next post we will get into the creation of survey items, including deciding on a measurement scale and the proper wording of questions.



Follow us on Twitter @nait_ir