Water Purification

Overview

ImpactMatters generates estimates of impact — estimates that quantify the causal effect nonprofits have on social outcomes relative to cost. For example: a nonprofit provides a year of clean water to one person for an average of $20. Our estimates incorporate best principles in social science, described in our Impact Methodology.

This document describes our methodology for estimating the cost-effectiveness of water purification programs. We define water purification as the processes of filtration or disinfection to ensure that water used by households for drinking, cooking, personal hygiene and other domestic uses satisfies the criteria for “improved” established by the Joint Monitoring Programme for Water Supply and Sanitation.1

In this document, we describe water purification as an intervention; the outcome by which we measure the impact of water purification programs; our methodology for estimating their cost-effectiveness; and a checklist of data required from nonprofits to calculate cost-effectiveness.


Description

In 2015, 663 million people lacked access to “improved” drinking water sources, and even more lacked access to “safely managed” water.2 The World Health Organization (W.H.O.) estimates that safe water could prevent 1.4 million child deaths from diarrhea, 500,000 deaths from malaria and 860,000 child deaths from malnutrition each year.3 Furthermore, the act of collecting clean water is a major burden in many developing countries. In some Sub-Saharan African countries, close to half of all households report spending over 30 minutes per trip to collect water.4 The burden falls disproportionately on women and girls, who are responsible for water collection in eight out of 10 households with water off premises.

The appropriate water purification program for a given community depends on factors like the quality of source water, volume of water to be treated, user preferences and the burden of maintenance. Because many solutions require behavior change on the part of the user and device upkeep, diligent monitoring of initial uptake, continued use and device effectiveness over time is particularly important for water purification programs. Some of the most common purification techniques include:5 biosand filtration; membrane filtration (micro-, ultra- and nanofiltration, depending on pore size); chlorine tablets; ceramic pot or ceramic candle filtration; and solar disinfection.

We use the term “water purifier” broadly. Some water purification systems physically filter out unwanted particles while others chemically disinfect water to kill harmful microorganisms. We refer to all of these technologies as “water purifiers.”


Outcome

A year of clean water for one person
Final outcomes: Improved health and time saved

We measure the success of water purification programs as their cost to provide a year of clean water to one person. Clean water improves the lives of beneficiaries in two primary ways: better health and time saved that otherwise would have been spent collecting water. We choose “a year of clean water” as the metric of analysis because it is closely correlated with these two final outcomes.


Methodology for estimating attributable outcomes

In algebra, we calculate the attributable outcomes of a water purification program (O) as follows:

O = A * B * C * D * (1 - E) * F
where:
A = Number of water purifiers distributed
B = Number of people served by each water purifier
C = Uptake; percentage of people, B, who regularly use the water purifier
D = Efficacy rate of water purifier
E = Counterfactual increase in clean water access
F = Number of years the water purifier remains in use

We first record the number of water purifiers distributed (variable A), e.g., 1,000 household water purifiers distributed to families in rural Ecuador in 2017.

If the nonprofit has specified how many people live in each household, we use its figure as variable B. If this information is unavailable but we know that each purifier serves one household, we make an assumption based on publicly available statistics on the average household size in the country or countries in question, e.g., 3.8 average members per household in Ecuador.6 We then multiply variables A and B to calculate the number of people receiving water purifiers from the nonprofit: 1,000 water purifiers * 3.8 people served per purifier = 3,800 people.

Next, we adjust by the percentage of people who actually use the water purifier. The success of household water purification interventions is highly dependent on a change in behavior on the part of beneficiaries; purifiers that are impractical or otherwise poorly designed can suffer from low uptake, resulting in low impact. If the nonprofit has reported the uptake rate among its beneficiaries, we use that rate as variable C. If this information is not available, we first search for the type of purifier distributed by the nonprofit. The three most common types tend to be: biosand filters with a plastic body; biosand filters with a concrete body; and hollow fiber membrane filters like the Sawyer PointONE filter. We then use average uptake rates (measured one year after distribution) observed in field studies: 88 percent for plastic biosand filters7; 90 percent for concrete biosand filters8; and 50 percent for hollow fiber membrane filters.9 For example: 3,800 people * 90 percent uptake = 3,420 people who use the concrete biosand purifiers distributed by a nonprofit. [If no purifier type has been specified, we take a simple average of these uptake rates.]

We define the efficacy rate of the purifier as the overall bacterial removal efficiency; where this rate is not available, we use the E. coli removal rate specifically. This variable is rarely reported by the nonprofit, so where necessary, we substitute figures from field studies for variable D: 92 percent for plastic and concrete biosand filters and 90 percent for hollow fiber membrane filters. For example: 3,420 people * 92% efficacy rate = 3,146 people obtain clean water from their concrete biosand filters.

Next, we estimate the percent of people who would have gained access to clean water without the nonprofit (the “counterfactual”). We do so to correctly attribute to the nonprofit only those who gain access to clean water as a result of its intervention. We draw on data maintained by the Joint Monitoring Programme for Water Supply, Sanitation and Hygiene (J.M.P.)10 on the level of coverage of household drinking water services in each country, disaggregated by rural and urban areas. We calculate the annual change in the coverage of “safely managed” drinking water for the country or countries where the nonprofit operates. We assume that this annual change represents the percent of the nonprofit’s beneficiaries who would have gained access to clean water in the absence of its intervention. In other words, beneficiaries are assumed to gain access to clean water at the same rate as others in the country. For example, the coverage of “safely managed” drinking water services in rural Ecuador was 55.44 percent in 2010 and 56.91 percent in 2015 — or an average annual increase in coverage of 0.29 percentage points (variable E). We calculate attributable outcomes as follows: 3,146 * (1 - 0.0029) = 3,137 people gained access to clean water as a result of the nonprofit’s intervention.

Finally, we estimate the number of years the purifier remains in use. If the nonprofit reports this data, we use it as variable F. If it does not, we make an assumption based on the type of purifier it distributes. Field studies show that plastic biosand filters remain in use for an average of five years11; concrete biosand filters for 10 years12,13; and hollow fiber membrane filters for one year.14 For example: 3,137 people * 10 years = 31,370 person-years of clean water access.


Methodology for estimating cost

Below, we summarize the most important aspects of our methodology for estimating the costs of water purification programs. For a detailed discussion of what sources of data we use, how we treat specific line items and accommodate variation in accounting practices, see Reference Manual on Data Analysis.

Costs we include

ImpactMatters estimates cost-effectiveness from the perspective of a socially minded donor. This means we count all important costs associated with a program regardless of who incurs them. Generally, the key cost-bearing parties are: the nonprofit itself; organizations with which it partners to run a program; the government (taxpayers); and the nonprofit’s beneficiaries.

Nonprofit costs

We include the total cost of the nonprofit’s water purification program, including costs to manufacture or source purifiers, costs to store and distribute purifiers, and maintenance costs such as replacement parts. Total program cost often also includes costs to train beneficiaries to both use and maintain their own purifiers.

Many nonprofits running water purification programs also offer other water, sanitation and hygiene (WASH) services, such as latrine construction and campaigns to improve handwashing among schoolchildren. In these cases, we attempt to isolate the costs specific to water purification.

Beneficiary costs

Some nonprofits charge households a small fee for their purifiers. We deduct this revenue from the nonprofit’s program cost, then add this revenue to beneficiary costs.


Methodology for calculating impact

To calculate the impact of a water purification program, we divide the total program-related costs incurred by all cost-bearing parties by the total person-years of clean water provided. Crucially, the numerator and denominator must match logically: The numerator reflects the costs incurred in generating the attributable outcomes reflected by the denominator.


Cost-effectiveness benchmarks

We determine whether a water purification program is cost-effective by asking whether there is a clear better alternative use of resources to achieve the same end. Specifically, we consider the following thought exercise: A nonprofit could spend programmatic resources to deliver clean water to a beneficiary or, alternatively, that beneficiary could purchase clean water on the market. The nonprofit’s program is therefore cost-effective if its cost to provide clean water is lower than the price a beneficiary would have to pay to obtain clean water on the market.

Our cost-effectiveness benchmarks come from a W.H.O. study on the cost of providing clean water in three regions of the world.15 Ideally, we would benchmark cost-effectiveness based on the cost a beneficiary would otherwise pay for the services the nonprofit provides, not the costs other providers would spend. For example, our benchmarks for food distribution programs are based on the cost a person would have to pay for a meal, not the cost to provide one.

Unfortunately, we were unable to find data on what market price beneficiaries would otherwise have to pay for clean water, and so we construct an estimated market price. To do this, we assume that the price paid by beneficiaries is simply the provider cost plus a markup. In the absence of data to guide this adjustment, we mark up provider costs by a factor of 1.5. We acknowledge this is imperfect and will adjust our benchmarks as new data becomes available.

To calculate our benchmarks, we make minor adjustments to annualize the W.H.O figures and convert them into 2019 dollars. . We then apply the 50 percent markup to set the following regional benchmarks for the per capita cost of a year of clean water:

Table 1

Region

Cost/year of clean water

Latin America and the Caribbean

$18.10

Africa

$11.43

Asia

$15.85

Following ImpactMatters conventions for services that are life-critical (e.g., emergency food distribution, clean water), we set benchmarks such that programs receive 5 stars if they provide water for less than 75 percent of the estimated market price and 4 stars if they do so for less than 125 percent.


Nonprofit checklist of data needed to calculate impact

The following data is necessary to estimate the impact of water purification programs.

Table 2

Checklist item

Required from nonprofit?

Details

Program activities

Yes

A program is a set of goods or services provided by the nonprofit to a population of beneficiaries with the goal of improving one or more outcomes. Generally, a program consists of the same components delivered to each beneficiary, with only minor deviations across different settings.

Geography

Yes

We recommend nonprofits report the number of water purifiers distributed by country and whether recipients live in rural or urban areas. This allows us to use country-specific data to estimate the counterfactual.

Timeframe

Yes

We recommend nonprofits report annual figures that align with their fiscal year.

Type of water purifier distributed

No

The type of water purifier is used to determine water purification efficacy rate, uptake by beneficiaries and years purifiers continue to be in use. Examples include biosand filters and hollow fiber membrane filters (like the Sawyer PointONE).

Number of water purifiers distributed

Yes

We recommend nonprofits report the number of water purifiers distributed in each country separately.

Number of people served per purifier

Yes

We recommend reporting the number of people, not just the number of households or schools, who receive their purifiers.

Uptake of purifiers

No

We recommend tracking the percent of beneficiaries who actually take up the purifiers and specifying when this uptake rate was measured (in terms of the number of months or years since distribution).

Efficacy rate of purifier

No

If not reported by the nonprofit, we use estimates from the research literature of efficacy rates associated with different types of purifiers.

Counterfactual increase in access to clean water

No

If not reported by the nonprofit, we use the change in coverage of “safely managed” drinking services, as defined by the J.M.P., in the country or countries where the nonprofit’s program operates.

Years purifier is in use

No

If not reported by the nonprofit, we use estimates from the research literature of the duration of continued use for different types of purifiers.

Program cost

Yes

We recommend reporting total costs, including costs paid out of pocket by volunteers.

Beneficiary cost

No

We recommend reporting participant costs if they are substantial. They can be estimated at $0 if they are not substantial. Some nonprofits charge households a small fee for their purifiers. We deduct this revenue from the nonprofit’s program cost, then add this revenue to beneficiary costs.

Partner cost

No

We recommend reporting partner costs if they are substantial. They can be estimated at $0 if they are not substantial. Partner costs are infrequently reported and are defaulted to $0 for water purification campaigns.

Limitations of our analysis

Oversimplification of the outcome

Using “years of clean water provided” as the metric of analysis does not capture other dimensions of accessing clean water, such as the quantity of water and ease with which it is accessed.

Cost of reaching special populations

Some nonprofits may have to incur additional costs to reach particularly disadvantaged patients. For instance, a nonprofit may have to spend more to transport both its staff and water purification equipment to remote locations.

Cost of reaching special locations

The same goods and services may be priced and taxed differently in different locations, making it more or less costly to run an identical program depending on where the nonprofit chooses to work. This is, in part, accounted for by our region-specific cost-effectiveness benchmarks of the market price to obtain clean water in Latin America and the Caribbean, Africa and Asia.

Specific counterfactuals

To understand the impact of a program, we ask the counterfactual question: What would have happened to beneficiaries if the program had not, counter to fact, been there to serve them? Because the vast majority of nonprofits have not conducted impact evaluations, we need to construct our own counterfactuals based on public data sources and the research literature. But in doing so, we risk masking variation in effectiveness across nonprofits. For instance, under our methodology, any nonprofit working in the same rural or urban part of a country faces the same counterfactual assumption (the percent of people who otherwise would have gained access to clean water). Nonprofit A might specifically seek out those on the margins of water service systems — and in so doing, expend substantial resources on, for instance, needs assessments to channel its resources where they are most needed. Meanwhile, Nonprofit B may work in a neighboring town with existing and increasing access to water services. But lacking data on counterfactual access specific to each nonprofit, we apply the same counterfactual in both cases, masking the variation in their targeting.

Data quality

Our estimates rely on data made public by nonprofits on their websites, annual reports, financial statements and Form 990s. There are, of course, ambiguities in the data and our interpretation of the data may not always match the nonprofit’s intention. For instance, we find most nonprofits do not report the efficacy of their own water purifiers, so we must draw on average efficacy rates from field studies elsewhere. If a nonprofit outperforms or underperforms field averages, our cost-effectiveness estimate will likely be an underestimate or overestimate, respectively. For more detail on our sources of data and how we interpret them, please see Reference Manual on Data Analysis.

Representativeness of (analyzed) programs

We only issue ratings for nonprofits if we can perform analysis on 15 percent or more of the nonprofit’s total program budget. This approach means some nonprofits are rated on only some of their programs. The remaining programs, which we could not analyze, could be more or less cost-effective than the programs we analyzed.




Footnotes

1

Improved drinking water sources are defined by the W.H.O./UNICEF Joint Monitoring Programme for Water Supply, Sanitation and Hygiene as those that “have the potential to deliver safe water by nature of their design and construction, and include: piped water, boreholes or tubewells, protected dug wells, protected springs, rainwater, and packaged or delivered water.”                                                                                                                                                                                                

2

Safely Managed Drinking Water - Thematic Report on Drinking Water 2017                                                                                                                                                                                                

3

W.H.O. (2008) How does safe water impact global health?                                                                                                                                                                                                

4

Safely Managed Drinking Water - Thematic Report on Drinking Water 2017                                                                                                                                                                                                

5

Household Water Treatment and Safe Storage Fact Sheets (Detailed)                                                                                                                                                                                                

6

We consult statistics such as those reported by the United Nations’ Household Size and Composition Around the World.                                                                                                                                                                                                

7

The Water and Sanitation Program: Improving Household Drinking Water Quality: Use of BioSand Filters in Cambodia                                                                                                                                                                                                

8

Aiken et al. (2011) An Assessment of Continued Use and Health Impact of the Concrete Biosand Filter in Bonao, Dominican Republic                                                                                                                                                                                                

9

Murray et al. (2015) Laboratory Efficacy and Field Effectiveness of Hollow Membrane Fiber Microfilters Used for Household Water Treatment in Honduras                                                                                                                                                                                                

10

The J.M.P. is an initiative of the W.H.O. and the United Nations Children’s Fund (UNICEF).                                                                                                                                                                                                

11

The Water and Sanitation Program: Improving Household Drinking Water Quality: Use of BioSand Filters in Cambodia                                                                                                                                                                                                

12

Household Water Treatment and Safe Storage Fact Sheets (Detailed)                                                                                                                                                                                                

13

Sisson et al. (2013) Long-Term Field Performance of Biosand Filters in the Artibonite Valley, Haiti                                                                                                                                                                                                

14

Murray et al. (2015) Laboratory Efficacy and Field Effectiveness of Hollow Membrane Fiber Microfilters Used for Household Water Treatment in Honduras                                                                                                                                                                                                

15

W.H.O.: Regional and Global Costs of Attaining the Water Supply and Sanitation Target (Target 10) of the Millennium Development Goals