Selecting the Best Tools for the Job
This month our Associate, Kaitlyn, is also sharing some tips on selecting data collection methods.
When you're planning an evaluation, the world is (sort of) your oyster: you can choose from many different feedback options to gather people's opinions, experiences, thoughts, and concerns. Choosing the best feedback tool may not always be obvious, so here are some tips and tricks to make it a little easier.
Surveys—written questionnaires that allow you to collect both qualitative and quantitative data—are a great tool for lots of reasons. Standardized survey questions come in many different formats (i.e. multiple-choice, ranking, rating, select images, etc.). There are lots of free and paid survey tools available, so budget doesn't have to be a concern, and you can collect information online or on paper. Surveys can be relatively easy to analyze, and if you have a knack for numbers, you can conduct more advanced statistical analysis. If you’re looking for feedback on something specific—a program, an event, or a product—surveys are a great tool.
Surveys have one big bonus, and that’s quantification: the ability to collect accurate close-ended data, especially from a large number of people. Collect close-ended data by limiting people’s response choices: offer a simple yes/no, or a scale of one to five, or a predetermined set of answers. This makes it easy to perform data management and analysis on most survey platforms. Surveys can really help you with the question most organizations are trying to answer: What impact, and how much?
But surveys aren’t perfect. There are many ways you can unintentionally obtain biased answers, depending how you asked your questions and other factors. Some organizations treat surveys as an opportunity to take a referendum on everything, rather than just collecting what they need to know. The biggest downfall of surveys is that they are more limited in answering questions like why the impact happened (or didn’t), and how to improve, which means organizations may need to use other means to investigate those issues.
When to use a survey: When you’re talking to lots of people; when you have a strong focus to your questions; when you want to be able to easily analyze your responses; when responses need to be anonymous.
When to use something else: When you need to explore a sensitive or triggering topic with extra care; when you want to ask follow-up questions; when you’re working with a group with accessibility challenges like limited tech access or lower literacy levels (people living with low-income, kids, etc).
Interviews are one-on-one meetings with an informant to collect opinions, experiences, and feedback. They’re a fantastic, flexible tool to gain deeper understanding on a topic or program. Many interviews that I conduct are considered semi-structured: they’re populated with questions and topic areas to discuss, but interview subjects are encouraged to add in useful information or expand in unanticipated areas. (You can also use informal or standardized interview processes). Interviewing people allows you to observe their body language as well as their verbal responses, and you can ask more intimate question with this tool.
However, interviews can be lengthy and time-consuming to coordinate, conduct, and analyze. They can also require some special skills and training: both to become an unbiased empathetic interviewer, and to move through mountains of qualitative data afterwards!
When to use interviews: to gather deep knowledge on a topic or program; when you’re working with people who speak better than they read or write; when the topic is sensitive; when you need some flexibility in the questioning process; when you want to dig deeper into why/how your program did or did not achieve its results.
When to do something else: when you’re collecting many responses; when budget is limited; when you want to report numbers of statistics.
Focus groups gather information by assembling small groups of key people and providing them with prompts for discussion, which is recorded and analyzed. We typically think of focus groups in the context of product evaluation, but they can also be used when participants are reviewing their experiences or interactions with an organization or program. As with interviews, researchers can also observe expressions, tone of voice, and other cues. Focus groups are often less structured and more conversational than other research tools, allowing participants to riff on each other’s insights, which can be helpful in exploring undiscovered perspectives and building consensus and solutions at the group level.
However, focus groups do require some facilitation skills to ensure that all participants are contributing. You may also find participants influenced by other group members’ views, which can discourage outlier perspectives.
When to use focus groups: When you want a diversity of perspectives on a topic – (i.e. being able to interact with each other can provide more detailed and nuanced program improvement suggestions); when having a conversation will help participants achieve clarity.
When to do something else: when the topic is intimate or triggering; where there are a lot of topics to cover; when you there are power imbalances in the group and you want each person’s unbiased views.
It's also important to note that these aren't the only way you can receive feedback from your informants or stakeholders. Simple tools like polls on Instagram, Facebook, or during a Zoom meeting can collect quick responses in a timely way, while more complex tools like journals or journey mapping can give holistic insights. Tailoring your data collection tools to what will give you the best insights is a process that might take a bit of up-front work, but in the end, your evaluations will be stronger for it.