Author: Dr Clara Ong
5 mins read
Data collection is the make-or-break stage of every outcomes measurement journey.
It can be a hard slog if you do not have the right tooling and systems in place to support.
From my experience, some of the common reasons why outcomes measurement projects fail, or come to a premature halt, include:
- Unwillingness of beneficiaries to provide valid and accurate information
- Low response rates, leading to a lack of staff motivation to see out the process
- Lack of staff ownership
In this article, I will unpack each “data collection hurdle” and provide some deceptively simple tips on what I’ve seen to be effective in overcoming them.
1. Unwillingness of beneficiaries to provide accurate information
This often stems from a lack of trust by the beneficiary to provide honest feedback.
The beneficiary provides one of the most illuminating insights into how effective a program is. Unfortunately, their views and experiences are often overlooked and underrepresented.
Beneficiaries may be unwilling to provide honest feedback as they have little or no alternatives, possibly due to lack of access to alternative services, long waiting lists, etc. Therefore, they may fear losing the help they are getting if they were to provide critically honest feedback. It can be said that beneficiaries are more likely to accept a flawed intervention, than none at all. (Twersky, F., Buchanan, P., Threlfall, V. (2013) Listening to those who matter most, the beneficiary. Published on Stanford Social Innovation Review)
Tips on how to overcome this hurdle:
Building beneficiary trust is fundamental. A great starting point is to introduce a ritual of listening to the beneficiary, capturing their feedback through informal conversations such as “how do you think that went”, “we value your insights as it will help us to improve our services”.
Before sending a formal survey to your beneficiaries, speak to them in-person first about the objective of the survey, explain how their participation would help you to serve them better, and the efforts your organisation will undertake to ensure that any information they provide will be confidential and stored securely.
The nonprofit organisations we work with tend to get better response rates if they first built rapport and spoke to their beneficiaries before sending out a survey.
Another step is for organisations to close the feedback loop, by giving beneficiaries crucial feedback on how the information they have provided has resulted in real action. Even if there has been no action taken yet, providing feedback on what is on the roadmap is key to building trust.
2. Low response rates leading to staff fatigue
Low response rates can be the result of many factors, one of them being a lack of beneficiary trust as discussed above.
Another reason for low response rates could be due to demographic and environmental circumstances. For instance, your survey might be too complicated and difficult to understand, or it might not be culturally relevant.
Not providing enough information about the objectives of the survey, or a compelling reason as to why survey recipients should complete the survey, are further reasons for low response rates. Another obvious factor is having a survey that is long and unstructured.
Tips on how to overcome this hurdle:
A simple step in overcoming this hurdle is to provide plenty of information to your survey recipients on why they should complete your survey. Remember, it needs to be framed as what is the value to them, rather than to you and your organisation.
Keeping your survey short and simple is also important. If it takes more than 5 minutes to complete the survey, it’s probably too long. Keep questions brief and succinct, and as tempting as it is to include open-ended questions (where recipients provide free text responses), be wary that open-ended questions will add to your recipients’ effort and is likely to lead to non-completion rates.
I have found that a survey of less than 10 questions is optimum, with an 80% – 20% split of picklist – open-ended questions. Last but definitely not least, ensure that your survey is culturally appropriate for your cohort of survey recipients.
We see many organisations struggle with designing a survey that would entice high response rates. Creating a survey from a blank canvas is no easy feat. This is why I am a big believer of not reinventing the wheel. Rather, where possible use survey templates so that you are not starting from a nil base. Using standardised templates can also provide valuable benchmarking data.
Lastly, sending out survey reminders is a great way to boost your response rates. This is particularly effective if you are administering electronic/e-mail surveys. We have found that recipients are more likely to respond on the second or third survey reminder.
3. Lack of staff ownership
Enlisting the support and buy-in from frontline staff is crucial to keeping the data collection process alive. Not having staff ownership will bring your organisation’s outcomes journey to a screeching halt.
Your staff can choose not to be supportive of the process for several reasons –
- Lack of capacity and time
- Not believing in the value of outcomes measurement
- Not understanding why measure outcomes
- Finding it a chore, due to the diversity of funder reporting requirements already imposed upon their workloads
Tips on how to overcome this hurdle:
First, take the time to understand the reasons why your staff is not supportive. Start by having a conversation with them to find out what their perceived barriers are, and why they might be sceptical of the process. While their perceptions may not always be indicative of reality, these perceptions drive behaviour so it is important to listen.
A lack of capacity and time, and the perception of data collection (or outcomes measurement more generally) being a chore, might be addressed by investing in smart tooling and systems to automate the end-to-end data collection process from survey scheduling, send-out and reminders, through to data entry, collation and analysis.
I have seen many organisations fail at their data collection efforts because they have tried to do so manually, perhaps through using paper surveys. Then there are organisations who recognise that manual effort is not sustainable, and try to pull together different tooling and systems (for example, electronic survey platforms with Excel) that often end up being more cumbersome than efficient.
There are a number of great tooling options available to help automate and streamline the data collection, hence simplifying the entire process. For example, the Socialsuite platform automates data collection by automatically sending out follow-up surveys and reminders to beneficiaries with minimal manual effect.
If your organisation has made a commitment to outcomes measurement, then it needs to seed an outcomes-driven culture by folding outcomes into its internal and public-facing narrative. Also important is investing in building staff capabilities to help them understand the value of outcomes measurement. There are workshops, webinars, tools and resources that can kickstart this process.
Download our toolkit to learn how to get your organisation ready for outcomes measurement
I hope this article has helped shed guidance on how to overcome some of the common data collection hurdles. In summary, take the time and listen to your staff, reflect upon how you can build beneficiary trust and whether your surveys are appropriate for your intended recipients. Last but not least, consider using proven survey templates and tooling.