An impact estimate captures attributable outcomes relative to cost. The numerator of the impact ratio is our estimate of the outcomes caused by the philanthropic intervention. The denominator of the ratio is our estimate of the cost of generating the change in primary outcome. The ratio of outcomes to cost captures the dollar cost of generating a one unit change in primary outcome. Example: cost of saving a child’s life; the cost of graduating one more student from high school; the cost of boosting household income by $100 through a microfinance intervention. At the very least, estimation of impact allows for comparing the performance of similar interventions. For example, which of several medical interventions costs the least to save one life; which of several high schools costs the least to successfully graduate a student. And so on. Said another way, impact estimates provide a powerful means by which to judge a nonprofit’s success in carrying out its mission through the program under analysis.
Programs aim to generate outcomes for their beneficiaries. To calculate the outcomes produced by a program, we subtract counterfactual outcomes from observed outcomes. Observed outcomes are the changes from the time of entry to a program to sometime after completion. Counterfactual outcomes are the changes that would have occurred had participants not in fact participated in the program. Some participants may have succeeded (as measured using a program’s indicators of success) in the absence of this or any other program. And some may have received different, comparable services from other sources.
Counterfactual success can be estimated but cannot be directly observed. Social scientists have developed numerous techniques to estimate counterfactual outcomes. There is a hierarchy of rigor among the techniques and an extensive literature on methodological limitations and their potential implications. However, the most rigorous technique may not always be feasible to employ. Ultimately, the choice of technique depends on factors like the nature of an intervention, data availability, budget and the context in which a nonprofit is operating.
Ideally, nonprofits develop strategies for estimating counterfactual success. The strategies might include: (i) analyzing data on individuals serving as comparison group in studies of similar interventions; (ii) analyzing data on participants prior to their entering the program under review.
In practice, many nonprofits do not think about counterfactual estimates and don’t know where to start. In that circumstance, ImpactMatters develops a “best guess” estimate of counterfactual success. When possible, we turn to the research literature. Though there may not be an experimental or quasi-experimental study of the program under our review, there may be studies of similar programs in similar contexts that can serve as a basis for estimation. All else the same, we use experimental studies whenever available. If there are no suitable randomized controlled trials, we look for strong quasi-experimental studies. Examples: comparative interrupted time series, regression discontinuity design, or difference-in-differences with matched comparison or synthetic control. If those are not available, we look for weak quasi-experimental approaches. Examples: matched comparison by itself. We then turn to non-experimental (observational) studies. There are times when other considerations such as sample size, age of the study and relevance of the population and context (i.e., external validity) mean that a study with a less statistically rigorous method is more appropriate (i.e., all else is not the same). This requires a judgment call.
Estimating Impact with External Evidence¶
In estimating program impact, we routinely start with program data provided by the nonprofit under review. We label as “internal evidence” data provided by the nonprofit about its program, studies conducted on the program by the nonprofit itself or studies by independent evaluators. We label as “external evidence” published data from studies conducted on the same type of program or on similar programs in a different setting.
In addition to providing a best guess for counterfactual estimates, described above, we use external evidence in two ways:
Linkage of intermediate outcomes and outputs to unobserved, longer-term mission-driven outcomes: Internal evidence might provide impact estimates for intermediate outcomes or a count of outputs. In this case, we turn to external evidence (literature reviews, including meta-analysis) to establish a causal chain, linking intermediate outcomes or outputs to primary outcomes.
Extrapolating impact trajectory: Nonprofits may collect insufficient data on participants after treatment. As a result, our impact measures assume that impact decays at a rate dependent upon the outcome under review. The speed and shape (for example, linear) of the decay is based on evidence from external literature. If evidence on duration is limited but the intervention would appear to have extended impact, we assume impact decays linearly over a two-year period (see Extending Impact below).
In programs for which multiple intermediate outcomes lead to the same primary outcome, complicated interactions can make difficult the estimation of impact. In these instances, the analysis will note whether external studies provide a statistical basis for parceling out impact on a primary outcome among multiple intermediate outcomes. The statistical tools most commonly used for this purpose include regression and factor analysis. In the absence of such studies, we incorporate the impact on primary outcomes of intermediate outcomes, but, to avoid double counting, we reduce the impact of each intermediate outcome on primary outcomes by 10 percent.
Discounting of Future Benefits¶
Taking account of the gap between the time that a participant receives an intervention and the time that the participant’s behavior and outcomes would be expected to occur raises the question of discounting. In general, social scientists assume that a $1 benefit received in year 10 is worth less than a $1 benefit received in year 1. And a cost of $1 incurred in year 10 is worth a fraction as much as a cost of $1 incurred today. Discounting takes account of the time differences between when costs are incurred, and benefits are received. We use a 5 percent discount rate, the rate employed by the World Bank and International Monetary Fund.1 We assume this discount rate also accounts for inflation over time.
Third-Party and Other Effects¶
Third-party effects, also known as externalities, are costs or benefits borne by non-participating individuals. For example, a job training program might benefit participants but harm non-participants by creating short-run competition for available positions and by reducing wage growth in the long run.
Externalities are excluded from the outcomes and cost analysis given the difficulty in reliably estimating them. However, where possible, we include a qualitative analysis of plausible externalities, with reference to the literature, including those that may become relevant if the nonprofit operates the program at a larger scale.
Other effects we discuss, if relevant, include downstream byproducts of primary outcomes. For example, a program that produces large effects on household income, its mission-driven outcome, may also generate downstream effects on material hardship or housing stability.
Handling Multiple Outcomes¶
Impact estimates are calculated for each outcome separately. The attribution of costs to separate outcomes poses a challenge. As best as data allow, we count only those costs that apply to the primary outcome under review. Which costs — especially shared costs, often referred to as admin costs — rise and fall as a primary outcome rises and falls requires guesswork as well as detailed observation. In all such cases, we state clearly what assumptions we have made.
To provide context for the size of impact estimate, we compare the estimate to a benchmark for that outcome set based on convention or market prices. If the impact estimate beats the benchmark, we conclude the program is cost-effective.
For some programs, we expect outcomes to last longer than the period observable. For example, a program that increases the number of students who graduate from high school should generate long term increases in outcomes. For these programs, we use assumptions and research literature to determine how long to extend outcomes.
If a nonprofit is likely generating outcomes post-program, our basic assumption is to extend outcomes for two years following the last observed date. The outcomes decline each year, so the first year is two-thirds the annual impact and the second year is one-third the annual impact. We apply this extension across the board — for example, all programs of a certain type to boost employment would receive the same bump.
For certain types of interventions, we may apply a longer outcomes extension if there is compelling research literature to suggest that a longer extension is warranted.
We calculate the costs associated with generating the estimated outcomes from the perspective of a socially minded donor. This means we include costs borne by the nonprofit, beneficiaries and partners, including government. Our data also enables us to calculate impact from other perspectives (for example, the nonprofit or government agencies).
Beneficiaries’ costs – costs borne directly by participants – include travel, materials and tuition. The opportunity cost to participants of lost wages will be included, if applicable.
Marginal and Fixed Costs in Impact Estimates¶
We ideally calculate impact on the basis of marginal costs of expanding the program to achieve an extra unit of success. The definition of a unit of success depends on the type of intervention. For example, for a job training program this may be an additional trainee who finds permanent work.
However, rarely can marginal and fixed costs be distinguished. We therefore focus on program costs, and exclude the costs of fundraising and management, which are assumed to be fixed with an incremental programmatic expansion. For small and medium donations, we believe this approximates to a reasonable degree the actual use of dollars by a charity. Following Meer’s analysis, each additional dollar will likely not be allocated to fundraising and management, or at least not at the same rate as the average ratio of fundraising and management to total costs.
Valuing Non-monetary Costs¶
We generally assign a cost of $0 to any donated goods on the assumption that they would otherwise have gone to waste. However, if there is reason to believe the goods still have market value (i.e., could still be sold at a non-zero price), we will count the fair market value of those goods.
We generally assign a cost of $0 to any substantial contributions of skilled or unskilled volunteer time spent on activities central to the nonprofit’s mission. This is because we believe volunteers benefit in important non-monetary ways — fulfillment, for instance, in supporting the cause.
Note that the Form 990 — unlike audited financial statements — excludes donated services from revenue and expense totals.
Some nonprofits generate revenue from their activities. For example, a food bank may sell some of its unused food inventory to non-beneficiaries. In cases like this, we deduct program revenue from program costs because that revenue offsets the cost of running the program.
For programs that take place outside the U.S., currencies are converted into U.S. dollars using current (at the time of reporting) foreign exchange rates. We do not adjust for purchasing power of a dollar because nonprofits use donations to buy the real basket of goods, not a price-adjusted basket.
The World Bank, “Staff guidance note on the application of the joint bank-fund debt sustainability framework for low-income countries” (82566, The World Bank, 2013), pp. 1–64.