Data collection is the make-or-break stage of every outcomes measurement journey.
It can be a hard slog if you do not have the right tooling and systems in place to support.
From my experience, some of the common reasons why outcomes measurement projects fail, or come to a premature halt, include:
- Unwillingness of beneficiaries to provide valid and accurate information
- Low response rates, leading to a lack of staff motivation to see out the process
- Lack of staff ownership
In this article, I will unpack each “data collection hurdle” and provide some deceptively simple tips on what I’ve seen to be effective in overcoming them.
1. Unwillingness of beneficiaries to provide accurate information
This often stems from a lack of trust by the beneficiary to provide honest feedback and the inputs that is required from them.
The beneficiary data source provides one of the most illuminating insights into how effective a social service or program is. Unfortunately, their views and experiences are often overlooked and underrepresented.
Beneficiaries may be unwilling to provide honest feedback as they have little or no alternatives, possibly due to lack of access to alternative services, long waiting lists, etc. Therefore, they may fear losing the help they are getting if they were to provide critically honest feedback. It can be said that beneficiaries are more likely to accept a flawed intervention, than none at all (Twersky, F., Buchanan, P., Threlfall, V. (2013) Listening to those who matter most, the beneficiary. Published on Stanford Social Innovation Review).
Tips on how to overcome this hurdle:
Building beneficiary trust is fundamental. A great starting point is to introduce a ritual of listening to the beneficiary, capturing their feedback through informal conversations such as “how do you think that went”, “we value your insights as it will help us to improve our services”.
Before sending a formal survey to your beneficiaries, speak to them in-person first about the objective of the survey, explain how their participation would help you to serve them better, and the efforts your organization will undertake to ensure that any information they provide will be confidential and stored securely.
The nonprofit organizations we work with tend to get better response rates if they first built rapport and spoke to their beneficiaries before sending out a survey.
Another step is for organizations to close the feedback loop, by giving beneficiaries crucial feedback on how the information they have provided has resulted in real action. Even if there has been no action taken yet, providing feedback on what is on the roadmap is key to building trust.
2. Low response rates leading to staff fatigue
Low response rates can be the result of many factors, one of them being a lack of beneficiary trust as discussed above.
Another reason for low response rates could be due to your survey recipients’ demographic and environmental circumstances. For instance, the survey might be too complicated and difficult to understand; or it might not be culturally relevant.
Not providing enough information about the objectives of the survey, or a compelling reason as to why survey recipients should complete the survey, are further reasons for low response rates. Another obvious factor is having a survey that is long and unstructured.
Tips on how to overcome this hurdle:
A simple step in overcoming this hurdle is to provide plenty of information to your survey recipients on why they should help you to complete the survey. Remember, it needs to be framed as what is the value to them, rather than to you and your organization.
Keeping your survey short and simple is also important. If it takes more than 5 minutes to complete the survey, it’s probably too long. Keep questions brief and succinct, and as tempting as it is to include open-ended questions (where recipients provide free text responses), be wary that open-ended questions will add to your recipients’ effort and is likely to lead to non-completion rates.
I have found that a survey of less than 15 questions is optimum, with an 80% – 20% split of picklist - open-ended questions. Last but definitely not least, ensure that your survey is culturally appropriate for your cohort of survey recipients.
We see many organizations struggle with designing a survey that would entice high response rates. Creating a survey from a blank canvas is no easy feat. This is why I am a big believer of not reinventing the wheel - use survey templates as much as possible so you are not starting from a nil base. Using standardised measures can also provide valuable benchmarking data.
Lastly, sending out survey reminders is a great way to boost your response rates. This is particularly effective if you are administering electronic/e-mail surveys. We have found that recipients are more likely to respond on the second or third survey reminder.
3. Lack of staff ownership
Enlisting the support and buy-in from frontline staff is crucial to keeping the data collection process alive. Not having staff ownership will bring your organization's outcomes journey to a screeching halt.
Your staff can choose not to be supportive of the process for several reasons:
- Lack of capacity and time
- Not believing in the value of outcomes measurement
- Not understanding why measure outcomes
- Finding it a chore, due to the diversity of funder reporting requirements already imposed upon their workloads
Tips on how to overcome this hurdle
Understanding the reasons why your staff is not supportive is an important starting point. Start by having a conversation with them to find out what their perceived barriers are, and why they might be sceptical of the process. While their perceptions may not always be indicative of reality, these perceptions drive behaviour so it is important to listen.
A lack of capacity and time, and the perception of data collection (or outcomes measurement more generally) being a chore, might be addressed by investing in smart tooling and systems to automate the end-to-end data collection process from survey scheduling, send-out and reminders, through to data entry, collation and analysis.
I have seen many organizations fail at their data collection efforts because they have tried to do so manually, perhaps through using paper surveys. Then there are those organizations who recognise that manual effort is not sustainable, and try to pull together different tooling and systems (for example, electronic survey platforms with Excel) that often end up being more cumbersome than efficient.
There are a number of great tooling options available to help automate and streamline the data collection, hence simplifying the entire process and not adding to your staff’s workload. For example, the Socialsuite platform automates data collection by automatically sending out follow-up surveys and reminders to beneficiaries with minimal manual effect.
If your organization has made a commitment to outcomes measurement, then it is imperative that it seeds an outcomes-driven culture by folding outcomes into its internal and public-facing narrative. Also important is investing in building staff capabilities to help them understand the value of outcomes measurement. There are workshops, webinars, tools and resources that can help to kickstart this process.
Wrapping Up
I hope this article has helped shed guidance on how to overcome some of the common data collection hurdles that confront organizations on an outcomes journey.
So, take the time and listen to your staff, reflect upon how you can build beneficiary trust and whether your surveys are appropriate for your intended recipients. Last but not least, look at using proven survey templates and tooling before you decide to reinvent the wheel.
Over 90 public companies and 70 non-profit organizations use Socialsuite for tracking and reporting on their impact. With the help of our ESG software and expert team, businesses can easily get started on impact reporting, disclose faster, and save money compared to traditional methods. Whether you're new to impact reporting or looking to enhance your current practices, Socialsuite offers the tools and expertise needed to achieve your sustainability goals. Contact us to learn more about our solutions.
