When you are planning to survey your participants, staff, partners, or others, a key consideration is whether to collect identifiable or anonymous data.
Identifiable data means you can identify the specific individuals that provided certain answers. Asking for participants’ names is the most common way to identify people, but data can also become identifiable if you collect several demographics and their combination allows you to identify a specific person. For instance, you may only serve one male, Asian, that is above 75 years old, so asking about gender, ethnicity, and age would likely make your data partially identifiable.
Anonymous data means you cannot identify the specific individuals that provided certain answers. If you want to collect anonymous survey data, a good rule of thumb is to only collect demographics such that any combination of demographics would have at least 5 people matching those characteristics.
You might ask, “Which type of data should we collect?” The classic evaluator answer is, of course: “It depends.”
Based on my experience, I review the strengths of both options below and outline when you may want to use one versus the other.
Identifiable Data
Collecting identifiable data is very important in many program administration contexts. For example, knowing what people are allergic to, or collecting their emergency contact information, can have life-or-death impacts. Also, you may want to collect identifiable data if you would like to follow up with individuals about the data they provide you. For examples, businesses may want to contact dissatisfied customers to resolve issues, or doctors or program staff may change how they support or care for individuals depending on their survey answers.
For the purposes of evaluating the quality and effectiveness of programs, collecting identifiable data becomes absolutely necessary only when you want to track changes in individuals over time. You can read our other post for more details about conducting pre-post surveys and why it is important to be able to match individual’s pre and post survey answers.
Anonymous Data
Collecting unbiased identifiable data requires a culture of trust and open communication between you/your organization and the people you are collecting data from. For instance, collecting ethnicity is very important to monitor for systemic racism and to learn how to make programs and services more equitable. However, ethnicity and other sensitive data or feedback can be used in a discriminatory and harmful way. For example, people may be concerned that their feedback will be given less weight if they share that they make less income, or they may not want to share identifiable negative feedback about their boss if they are worried their boss may act in a retaliatory manner.
We have seen the trust factor seriously bias evaluation results; for example, when asking youth about risky behaviours at the start and at the end of a program. Youth will often minimize the extent of their risky behaviours at the start of a program; then, as trust between staff and youth increases, youth provide more honest answers near to the end of the program. In those types of situations, it is often better to collect data just at the end of the program, and ask about risky behaviours currently and what they were before the start of the program.
A Culture of Trust and Safety
You can take steps to help increase people’s trust when providing identifiable data. You should describe:
Why you are asking for identifiable data,
What you are (and aren’t) going to do with it, and
How you will protect it.
While providing those descriptions in your survey can help, it is more effective to establish ongoing trusting relationships and a culture where people feel safe to provide open and honest information.
As a success story, I worked with an organization on a staff survey, and they worked hard to create a culture of trust and open communication among their staff. I was concerned that making the survey identifiable would discourage people from providing critical and specific feedback, but the organization wanted the survey process to model their culture of trust and open communication. Sure enough, even though their responses were identifiable, staff at this organization provided more detailed and descriptive feedback than typical for surveys, and provided a strong mix of both positive and critical feedback.
Summary
Collect identifiable survey data when:
You have a culture of trust and open communication with respondents.
Knowing individuals’ characteristics is important to serve them better.
It is important that you can track individual’s changes over time.
Otherwise, collecting anonymous survey data will likely produce more accurate survey findings.
Comments