This manual describes the step-by-step procedures by which ImpactMatters analyzes water purification nonprofits. We recommend reading the program analysis methodology first.
We note the country in which the program operates, and whether they operate in rural areas, urban areas, or both, as this influences which counterfactual we use.
As a first source, we check the Form 990. Unfortunately, water purification 990s are not as uniformly formatted as, say, those of scholarship programs, so this information will not always be in the same place. Some common places we look are: Part I,1 - summary; Part III,1 - mission; Part III - line 4a.
If the information is not available on the 990, we turn to the nonprofit’s website. The website may have the information itself, or links to relevant annual reports. From here on, we refer to these locations as the “common areas” where relevant information may be found.
While country of operation is usually easily available, the urban/rural distinction may not be. We search for the town on google and use our judgment as to whether it has frequently been described as a “city,” or use images to determine if it’s rural.
A note on multiple countries: Often, a nonprofit that provides water purification services will have programs in multiple countries. If this is the case, and data is available for the different programs in different countries, we will record data for each country-program separately (e.g., Pure Water for the World works in rural Honduras, and rural and urban Haiti). Since different countries use different counterfactual rates, we fill in data “horizontally” (one row each) for each individual country-program, and then merge the relevant impact values into a “total” row.
Type of purification system¶
The type of purification system is crucial information, as it informs the efficacy rates, uptake rates, and lifetime of the water purifier — all key variables in our impact calculation. The most common purifier types we have found are:1
Plastic biosand filters
Concrete biosand filters
Hollow fiber membrane filters (e.g., Sawyer PointONE)
Like program location, the source of this information is not standardized, but can be found in one of the common areas (see above).
Note that the purifier types will not always be one of the three above, but whatever the type may be, we note it. If a nonprofit uses a purifier type other than the three above (some nonprofits use specific, custom filters, for example), we take the average of the relevant efficacy rates, uptake rates, and lifetimes of the purifier types for which we do have data, and use that average value in our calculation.
Note also that since different purifier types lead to different impact calculations, if a nonprofit uses different purifiers in different programs, we separate them out like we would programs in different countries.
Number of purifiers distributed¶
Most often, this information is found in the 990. When the information is not available in the 990, we turn to the rest of the common areas, looking for some report that gives us the number of purifiers the nonprofit distributes.
Note that if we are evaluating a program in just one country when a nonprofit operates in several, we must take extra care to ensure we are only capturing the purifiers distributed in our country/program of interest.
If disaggregated data is not available, we won’t evaluate the nonprofit.
If no data is available on the number of purifiers distributed, we do not evaluate the nonprofit.
Average number of people served by a purifier¶
When this data is directly reported by the nonprofit for the program we are evaluating (in the common areas, if anywhere), we simply take that number.
Often nonprofits will report the total number of purifiers distributed, and the total number of people served, in which case a simple calculation gives us the average number of people served per purifier.
Sometimes, the calculation requires more assumptions:
If a nonprofit reports that a purifier services one “family,” we take the average family size for that country as reported by the U.N. or other global data sources.
If a nonprofit reports a purifier serves a “community,” we try to track down the community to which they are referring and use that population size. An example: Start with One International sent a tweet saying “Using leadership to provide #cleanwater to a community of over 50k people of #Lanet @Chiefkariuki @UZimaFilters,” so we used 50,000 as our relevant figure.
As with other categories, we ensure that we are only capturing the average number served by a purifier for the program in the country we are evaluating.
If disaggregated data is not available, we won’t evaluate the nonprofit.
If no data is available on the number of people served, we do not evaluate the nonprofit.
Years a purifier is in use¶
When a nonprofit reports for how long its purifiers remain in use (in one of the common areas), we simply take that number.
If a nonprofit does not directly report for how long one of its purifiers remains in use, we reference estimates from the research literature on how long different water purifiers last. For example, Sawyer PointONE filters have been observed to last in the field for an average of one year.
If a nonprofit does not provide a number of years in use, and does not use a purifier for which we have an estimate from the literature, we take an average of the lifespans of types of water purifiers from the literature and apply that value. This is imperfect, but not enough to disqualify a nonprofit. We hope that in the future, through our reporting process, nonprofits will be encouraged to report better data on the observed lifespan of their purifiers.
Note that we use observed device lifespans from field studies, not expected device lifespans from technical specification documents.
In the simplest case, when a nonprofit reports the relevant cost for a program (i.e., the cost associated solely with running the water purification program in the country we are evaluating), we simply take that value.
Note: If we are evaluating multiple programs across different countries, we don’t actually need the costs broken down by country. We ultimately calculate a single impact estimate for a nonprofit, even if it operates in multiple countries, and we only need to reference this total impact against a single total cost. We do, however, need to ensure that we are only capturing the total cost of the nonprofit’s water purification program, and not other services it may provide (see below).
Often, a nonprofit provides additional services along with the water purifiers themselves, such as sanitation and hygiene training, which complicates how we calculate our cost figure.
If a program includes additional services, but still breaks down costs, and provides a dollar value just for the portion of the program that distributes purifiers (or a dollar value of the additional services which we can subtract from a total cost), we take that cost.
If a program performs additional services and does not disaggregate costs, but a cost-per-purifier has been reported by the nonprofit, we will multiply the cost of a single purifier by the number of purifiers distributed.
If there is simply no way to separate the costs of additional services from the costs of providing water purifiers, we do not evaluate the nonprofit.
If a nonprofit reports any costs undertaken by partners (in the common areas) as part of its water purification program, we include them in our impact estimate.
Some nonprofits report beneficiaries purchasing or investing in water purification systems. When we know the value of the investment beneficiaries have to make (e.g., the cost of a purifier they need to buy), we do one of two things:
If we know how many purifiers are purchased, we multiply that number by the cost of the investment.
If we don’t know how many purifiers are purchased, we assume 50 percent of the total number of purifiers were purchased, and multiply that figure by the price per purifier.
If beneficiaries pay these fees directly to the nonprofit, we net them out of the nonprofit’s program costs. We do not subtract them from nonprofit costs if beneficiary fees are paid to some other entity (e.g., local water technicians).
To date, the nonprofits that have published enough data for an impact rating have all distributed water filtration systems, not chemical disinfection systems. We use the term “purifiers” to refer to both.