Introduction
In an era of increasing accountability and donor scrutiny, non-governmental organizations (NGOs) face mounting pressure to demonstrate that their programs actually work. Funding bodies — from governments to private foundations — want evidence, not just anecdotes. This has made impact measurement a cornerstone of modern development work, and at the heart of that measurement lies one deceptively simple tool: the survey.
Surveys allow NGOs to collect standardized data from the people they serve, track changes over time, and compare outcomes across groups. When designed and implemented well, they can reveal whether a health intervention reduced disease burden, whether a skills-training program improved livelihoods, or whether a community development project shifted social norms. But surveying is far from straightforward. It requires deliberate design, methodological rigor, and an honest reckoning with the limits of what questions can tell us.
What Is "Impact" and Why Does It Matter?
Before any survey can be designed, organizations must grapple with what they are actually trying to measure. In the development sector, "impact" has a specific meaning: it refers to the long-term, significant changes in people's lives that can be reasonably attributed to a program's activities. This is distinct from outputs (the number of workshops held or vaccines distributed) or outcomes (shorter-term changes in behavior or knowledge).
The distinction matters enormously. An NGO running a financial literacy program might easily count how many people attended classes (output) and how many passed a knowledge test (outcome). But whether participants' household savings actually increased, and whether that change was due to the training — rather than a good harvest season or a government cash transfer — is a much harder question. Surveys, when embedded within a robust measurement framework, are the primary tool for answering it.
Survey Types Used in Impact Measurement
NGOs typically employ several types of surveys depending on their program stage and evaluation goals.
Baseline Surveys are conducted before a program begins. They establish a snapshot of the target population's situation — their income levels, health status, knowledge, or attitudes — against which future changes can be compared. A well-designed baseline is arguably the most important investment an organization can make in its evaluation system.
Endline Surveys mirror the baseline and are administered at the program's close or at a specific point afterward. By comparing responses across the two time points, evaluators can assess whether and how much things changed.
Midline Surveys are deployed during program implementation. They serve as real-time feedback loops, allowing organizations to identify problems early and make course corrections before it is too late.
Longitudinal Surveys track the same individuals over an extended period — sometimes years — to understand whether program effects are sustained. These are resource-intensive but provide the richest picture of lasting change.
Control Group Surveys are used when NGOs employ experimental or quasi-experimental designs. By surveying both participants and a comparable group of non-participants, organizations can attempt to isolate the program's effect from other factors that might explain observed changes.
Designing Effective Survey Instruments
The quality of any evaluation is only as good as the questions asked. Poor survey design is one of the most common — and most consequential — failures in NGO impact measurement.
Good surveys begin with a clear theory of change: a logical map of how program activities are expected to lead to desired outcomes. Each survey question should correspond to a specific indicator tied to this theory. If an organization believes its nutrition education program works by increasing mothers' knowledge, which in turn changes feeding practices, which then improves child growth, then the survey must capture all three of these steps — not just the final anthropometric measurements.
Question design requires particular care. Leading questions, double-barreled items, and culturally inappropriate framing all introduce bias. For instance, asking "Don't you agree that handwashing is important?" will yield very different responses than a neutral behavioral observation. Similarly, questions about sensitive topics — household income, domestic violence, sexual health — require careful wording, sequencing, and often specialized protocols to elicit honest responses.
Translation and adaptation are critical in multilingual contexts. Direct translation of survey instruments rarely works; concepts must be back-translated and tested for equivalence of meaning. What "food insecurity" means to a researcher in Geneva may not resonate at all with a smallholder farmer in rural Uganda.
Pilot testing — running the survey with a small sample before full deployment — is not optional. It catches ambiguous wording, estimates the time required, and reveals practical issues that no desk review could anticipate.
Sampling Strategies
Who gets surveyed matters as much as what they are asked. NGOs must make deliberate decisions about sampling to ensure their results are meaningful and generalizable.
Random sampling gives every member of the target population an equal chance of being selected, producing results that can be extrapolated to the wider group. It is the gold standard but can be logistically difficult, especially in contexts with no reliable population registers.
Stratified sampling divides the population into subgroups (e.g., by gender, age, or geographic area) and samples from each. This ensures that smaller but important subgroups are adequately represented in the data.
Purposive or convenience sampling — selecting participants based on accessibility rather than probability — is common in resource-constrained settings but limits the ability to generalize findings. Organizations using these approaches should be explicit about their limitations.
Sample size is another critical variable. Too small a sample yields results that are statistically unreliable; too large a sample wastes resources. Power calculations, which estimate the minimum sample needed to detect a meaningful change, should guide these decisions from the outset.
Data Collection Methods
Traditional paper-based surveys are increasingly being replaced by digital tools. Mobile data collection platforms — such as KoBoToolbox, ODK, and SurveyCTO — allow enumerators to record responses on tablets or smartphones, with built-in validation rules that catch errors in the field. GPS coordinates can be embedded to verify location, and photos can be attached to individual records.
These technologies reduce transcription errors, accelerate data cleaning, and enable real-time monitoring of data quality. They have become essential infrastructure for serious impact measurement, even in remote or low-connectivity environments through offline functionality.
Computer-Assisted Telephone Interviewing (CATI) and online self-completion surveys have expanded access during and after the COVID-19 pandemic, though they introduce new challenges around sampling bias — particularly in populations with uneven access to phones or internet.
For sensitive topics, audio computer-assisted self-interview (ACASI) methods, where respondents listen to questions through headphones and enter responses privately, have shown promise in eliciting more accurate data on stigmatized behaviors.
Analysis and Attribution
Collecting data is only the beginning. The harder challenge is interpreting it honestly.
The most rigorous approach to establishing attribution — proving that observed changes were caused by the program — is the Randomized Controlled Trial (RCT), where participants are randomly assigned to receive the intervention or not. RCTs have produced some of the development sector's most influential findings, including insights from J-PAL (the Abdul Latif Jameel Poverty Action Lab) that have reshaped global health and education programming.
However, RCTs are expensive, time-consuming, and not always ethical or feasible. Many NGOs instead use quasi-experimental methods: difference-in-differences analysis (comparing changes in a treatment group to changes in a control group over time), regression discontinuity designs, or propensity score matching to construct comparable comparison groups.
Even without comparison groups, descriptive analysis of survey data can be valuable when interpreted carefully. Documenting the magnitude and distribution of changes, tracking against pre-set targets, and disaggregating results by gender, age, or other variables can surface important programmatic insights — as long as organizations are honest about what they can and cannot claim.
Common Pitfalls and Ethical Considerations
Several failure modes recur across the sector.
Confirmation bias can distort both survey design and analysis, particularly when evaluations are conducted internally by staff invested in positive results. Independent evaluation — commissioning external evaluators — offers a partial remedy, though it brings its own complexities.
Respondent burden and fatigue are real. Long, frequent surveys frustrate participants, reduce data quality, and can damage community trust. Brevity and necessity should guide every question included.
Survey ethics demand attention. Informed consent must be genuine — explained clearly in local languages, without coercion. Participant data must be stored securely and used only for stated purposes. In fragile or conflict-affected settings, surveys can expose respondents to risk if their answers could be identified, requiring careful anonymization protocols.
The question of who owns the data is increasingly central to ethical practice. Community data sovereignty frameworks argue that the populations surveyed should have meaningful input into how findings are used and shared.
Conclusion
Surveys, done well, are powerful instruments of accountability and learning. They help NGOs move beyond faith-based assumptions about their work toward evidence-informed practice — understanding not just that they are doing good, but whether that good is actually reaching the people who need it.
Done poorly, however, surveys can produce misleading findings that waste resources, distort program design, and ultimately harm the communities they are meant to serve. The difference lies in rigor: in thoughtful design, honest sampling, ethical implementation, and intellectually humble analysis.
As the sector continues to mature, the expectation is not merely that NGOs count what they do, but that they genuinely understand what their work accomplishes. Surveys are an indispensable part of that reckoning.
References
- Bamberger, M., Rugh, J., & Mabry, L. (2019). RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints (3rd ed.). SAGE Publications.
- Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. J. (2016). Impact Evaluation in Practice (2nd ed.). World Bank Publications. https://doi.org/10.1596/978-1-4648-0779-4
- Guijt, I. (2014). Participatory Approaches. Methodological Briefs: Impact Evaluation No. 5. UNICEF Office of Research – Innocenti. https://www.unicef-irc.org/publications/pdf/brief_5_participatoryapproaches_eng.pdf
- J-PAL (Abdul Latif Jameel Poverty Action Lab). (2022). Introduction to Evaluations. Massachusetts Institute of Technology. https://www.povertyactionlab.org/resource/introduction-evaluations
- Karlan, D., & Appel, J. (2011). More Than Good Intentions: How a New Economics Is Helping to Solve Global Poverty. Dutton.
- Muñiz Castillo, M. R., & Bangura, A. K. (Eds.). (2010). Human Development and the Comprehensive Development Framework. United Nations University Press.
- OECD DAC. (2019). Better Criteria for Better Evaluation: Revised Evaluation Criteria Definitions and Principles for Use. OECD Publishing. https://www.oecd.org/dac/evaluation/revised-evaluation-criteria-dec-2019.pdf
- Patton, M. Q. (2008). Utilization-Focused Evaluation (4th ed.). SAGE Publications.
- Ravallion, M. (2012). Fighting Poverty One Experiment at a Time: A Review of Abhijit Banerjee and Esther Duflo's Poor Economics. Journal of Economic Literature, 50(1), 103–114. https://doi.org/10.1257/jel.50.1.103
- White, H. (2010). A Contribution to Current Debates in Impact Evaluation. Evaluation, 16(2), 153–164. https://doi.org/10.1177/1356389010361562
- World Health Organization. (2016). WHO Guidance on Ethical Issues in Public Health Surveillance. WHO Press. https://www.who.int/publications/i/item/9789241549295
- Yin, R. K. (2018). Case Study Research and Applications: Design and Methods (6th ed.). SAGE Publications.
