For several years, I have partnered with United Way Halton Hamilton's ConnectED program to provide pro-bono 1:1 evaluation support to non-profits, and it has led to many learnings for everyone involved (based on our feedback survey responses, at least!). This article reflects on some of the learnings experienced through this partnership.
ConnectED is United Way Halton & Hamilton’s capacity building program for the social sector. Local non-profits can access quality, affordable, and accessible professional development and educational webinars. This blog post is part of the exploration to provide diverse resources that can expand the network's capacity more holistically, and that anyone can access. If you have feedback about capacity building for the social sector, you can contact Kirstin Webb, at kwebb@uwhh.ca.
My last ConnectED session with the Boys and Girls Club of Hamilton-Halton focused on collecting sensitive demographic data from individuals, including children and youth. Fortunately, many organizations are looking to collect more demographic data with a focus on Equity, Diversity, and Inclusion (EDI). Collecting a diversity of demographics and measuring how those demographics relate to experiences and outcomes can greatly aid us in our efforts to enhance EDI. As the saying goes, “what gets measured, gets done.”
Unfortunately, if organizations are not careful, collecting such data can be harmful, and individuals and equity-deserving groups have experienced such harm in the past. I have seen many organizations struggle to do this right, so I am happy to share some guidance based on my experience and knowledge of conducting evaluation in service of equity, diversity, and inclusion.
Before I share my knowledge, I would like to recognize that I am a student of BIPOC leaders in this space; some of which I link to below.
Minimize Harm
Research ethics direct us to minimize any potential harm to anyone we are collecting data from. Even when you are collecting data for administrative purposes, having a plan to minimize harm is still a best practice.
Plan How the Data Will Support Equity
You should not collect sensitive demographic data unless you have a clear plan and the capacity to use the data in a way that ensures equitable benefits to the various groups providing the data. This means using the data beyond just reporting statistics to funders or other stakeholders. It means having a plan for learning how to improve access and outcomes for equity-deserving groups, and having the capacity and commitment to make those improvements.
Privacy & Security
You should have a plan and systems in place to ensure:
People’s data is not shared or leaked to others who should not have access to it. This helps prevent data from being used in an unintended or harmful way.
Your own organization does not use the data to discriminate against, stigmatize, or treat clients in unwanted ways based on their demographics.
Question Design
You want to strive to ask questions in a way that affirms all identities rather than in a way that excludes or diminishes anyone, or makes them feel ‘othered’ or inferior. Here are two key tips for asking demographic questions in a way that minimizes potential harm:
Instead of “Other, please specify:” use “Another _________, please specify.” Unfortunately, all too often, registration forms and surveys still use the “Male,” “Female,” “Other” options when asking about gender. Labelling all non-binary gender identities as “Other” communicates that those individuals are “others.” The implied message is that they are not part of mainstream society and that their unique identities do not matter as much as binary identities.
When to use open-ended questions vs. checkboxes. A lot of aspects of people’s identities don’t fit neatly in checkboxes. When asking about characteristics that have a spectrum or a very large multitude of possible answers, it is best to provide an open-ended textbox, especially if you have not developed closed-ended options in collaboration with equity-deserving groups. For instance, there are hundreds of possible ethnicity answers. Including all possible options in a list will make the list too long, hard to navigate, reduce response rates, and can even create errors. However, shorter lists are bound to miss specific identities. As with the gender example, individuals whose specific identities are consistently not included on lists may begin to feel like their identities are invisible. It is better to simply ask, “what is your ethnicity?”, or “what is your gender?”and let people answer however they want. Asking open-ended demographic questions requires a bit of extra time to code the various responses into distinct categories for analysis. Ideally, that coding should be done with the input of equity-deserving groups. If you are going to use closed-ended options for questions about aspects like gender or ethnicity, then the best practice is to develop those options with the equity-deserving groups that you serve, as preferred language and categories changes across time, context, and groups.
Community-Led
A key principle of the Equitable Evaluation Framework is the orientation towards participant ownership. This principle means that you need to work with members from equity-deserving groups and they must have decision-making power. When collecting sensitive demographics, they need to be involved in:
Deciding if and when certain demographics are collected;
How to word questions and response options;
And, perhaps most importantly, how results should be interpreted and used.
Involving equity-deserving groups in those decisions increases the validity of the data and minimizes potential harm during the questioning and analysis stages.
Providing some sort of payment, benefit, or honorarium to those equity-deserving groups for their work and contributions is another important best practice that I follow. As an evaluator, I get paid a good wage, so I should not ask members of equity-deserving groups to do similar work for free.
Informed Consent
When collecting people’s data, it is best to get their informed consent/agreement for you to have and use it. That consent must be given voluntarily and without coercion. To ensure peoples’ consent is truly informed, you need to explain what is involved in sharing their information, and the positive or negative implications for deciding to provide their data or not. A good informed consent write-up should cover:
Why you are collecting the data. To help people feel comfortable providing you with their demographic data, explain that you are asking for the demographics because you understand that what works best for some people may not work best for others, and/or explain that you want to ensure different groups benefit equally from your programs.
What will be done: Typically, this involves noting what types of questions will be asked and about how long it will take to answer them.
Potential benefits of providing the data: Typical benefits include an improved program experience and outcomes for participants and others like them.
That providing the data is voluntary and choosing to not provide the data will not have negative consequences for the participant. Make it clear that participants can ask to un-share their data (i.e. have it deleted or returned to them).
How the findings will be used: Explain what decisions you hope to make with the data. If applicable, explain you will not share their individual data with others outside of your organization, but you will share grouped and summarized data in a report, and describe who you will share that report with.
Potential risks of providing the data: Some risks of providing demographic data include:
Being discriminated against, stigmatized, or treated in an unwanted way based on the provided demographics;
Demographic data being shared or leaked to others that should not have access to it and who may use it in an unintended or harmful way.
Best practice is to protect against those risks and explain:
How you will keep the data confidential.
Who will have access to the data and how results will be shared.
How you will ensure the data will not be used in an unintended or harmful way.
Writing up a good informed consent description protects you and your clients. It should also alleviate clients’ concerns about providing their data, which should improve response rates and encourage honest responses.
Resources
We All Count provides excellent guidance and resources on equity and data.
YouthREX and LGBT Youthline recently released a toolkit for asking about gender. The categories they provide should be appropriate in the near-future, but you should review the categories with some of the LGBTQ2S+ clients you serve.
This is a great three-page tip sheet for collecting information from people who may have experienced trauma.
Historically, compensating equity-deserving groups for their contributions to data collection design, analysis, and use has not been adequately budgeted for. The Patient-Centered Outcomes Research Institute’s participant compensation framework may help you devise and advocate for an appropriate budget-line when applying for grants.
Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans is lengthy, but it is what guides researchers in conducting ethical research, including guidance on minimizing harm and proper informed consent.