Tips for Reporting Impact
Good impact reporting means precisely and transparently sharing the difference you made in the lives of those you serve. It's a fundamental responsibility of nonprofits as stewards entrusted with not only donors’ gifts but also beneficiaries’ welfare. Carrying out this responsibility engages stakeholders at a deeper level, strengthening their commitment to your mission. Impact reporting is also an opportunity for honest, fact-based learning about what works and doesn’t work — so you can advance your mission even further next year.
Nonprofits often struggle to report impact well. We’ve rounded up the most common but non-obvious missteps of reporting impact. Below, we walk through seven tips to do it right.
Tip #1. One program at a time
Many nonprofits run multiple programs that serve different beneficiaries and aim to achieve different outcomes. Consider a neighborhood community center that provides college preparation and counseling to low-income high school students, free daycare services for children from low-income families and congregate meals for senior citizens. Articulating its impact on the community can feel like a daunting task.
The best place to start is with a single program. Programs are the building blocks that make up the total impact of your nonprofit. Each program is defined by three core ingredients: (1) a target beneficiary; (2) a desired change in an outcome for that beneficiary; and (3) a defined set of goods or services designed to achieve that desired change. Take, for instance, the college prep program. The target beneficiary is high school students from low-income families (ingredient 1). The desired outcome is higher earnings in adulthood as a result of graduating from college (ingredient 2, see more below). And the set of goods or services is preparation classes for college entrance exams and guidance on the application process from a trained counselor (ingredient 3). Treating the program as the unit of analysis is manageable and can bring greater clarity to the impact of your nonprofit as a whole.
Tip #2. Define the outcome
An outcome is a valuable success in the lives of beneficiaries. Consider again the neighborhood community center. The intended outcome of its free daycare services might be increased income, derived in two ways: parents save on daycare costs and can also earn more by working more hours. The intended outcome of the congregate meals program might be reduced hunger among food-insecure senior citizens in the community. For what an outcome is not, see our Methodology for Estimating Impact.
Defining fundamental outcomes guides behavior. Social change is demanding work and nonprofits are pulled in many different directions. It’s important to regularly assess whether the activities we devote our time and resources to are in service of our mission.
We recommend defining the one outcome each program aims to change. Designing a program to optimally achieve a single outcome is relatively straightforward. But designing a program to optimally achieve two or more outcomes invites onerous complexity.
On the surface, choosing a single outcome may seem like an oversimplification of the complex work nonprofits do. Besides having higher earnings, beneficiaries of the college prep program also go on to lead healthier lives, have higher savings for retirement and have better housing security. But digging a little deeper elucidates an important distinction between outcomes and downstream effects. Downstream effects are second-order changes triggered by a shift in the outcome. Because beneficiaries have higher earnings (the outcome), they have more money to spend on better health care and housing, and save for retirement (downstream effects). Focusing on shifts in a single outcome does not ignore downstream effects — they are in fact inherently wrapped up in the outcome because they are its byproducts.1
Tip #3. Be precise
Aspirational and emotive language is often used to describe the profound ways nonprofits affect the lives of beneficiaries, frequently in the form of anecdotes. While anecdotes and case studies are important in other ways,2 they are generally not sufficient basis for estimating impact. Impact estimation requires greater precision about the outcome you are changing and the magnitude of that change. For instance, “Our free daycare services eased the burden on 100 families this year” is vague. Reporting that you enabled 100 families to collectively save $10,000 in daycare costs this year is precise both in the outcome and the amount by which you changed it.
Tip #4. Model what you can’t measure
Often, what we can feasibly measure falls short of our desired outcomes. The neighborhood community center may not have the resources to survey former beneficiaries on their earnings over the course of their working lives, much less run a randomized evaluation to rigorously measure its impact.
Where measurement fails, modeling can often be a second best solution (sometimes first best, but that’s another story). Let’s say the community center is able to track whether former beneficiaries graduated from college. It can then use population statistics on the average earnings of college graduates compared to nongraduates to roughly estimate the boost in earnings owing to a college degree. Two important adjustments that would greatly improve this modeling: First, ensure that the average earnings data come from a population that is similar to beneficiaries. Second, ensure that graduates and nongraduates represented in the earnings data are similar to each other in all ways but college graduation itself; this helps combat the risk that other factors besides graduation are causing the difference you see in their earnings stats.
Modeling takes practice and some technical expertise. But the effort can yield a usable, transparent estimate of your impact where one did not exist before. It’s how we estimated the impact of over 280 postsecondary scholarship programs around the country to find the best for your dollar.
Tip #5. Come clean
Modeling, by its very nature, contains uncertainty. Even when assumptions are couched in the very best studies from the scientific research literature, some confidence interval will always exist around modeled estimates of impact. Being fully transparent about your methods helps donors, practitioners and raters like ImpactMatters not only follow along but also offer feedback. This does not have to take the form of a lengthy white paper or academic research. The research team at ImpactMatters has seen a growing wave of nonprofits annotating annual reports and websites with simple explanations of assumptions made and data sources that underlie each nonprofit’s statement of impact.
Tip #6. Counterfactual, counterfactual, counterfactual
We’ve written in the past about our obsession with the counterfactual and its centrality to impact analysis. Here is actionable advice on how to construct a counterfactual when an experimental control group is out of reach.
When randomized controlled trials are not feasible or appropriate, evaluations often take the form of quasi-experiments. A quasi-experimental evaluation tests a causal hypothesis by manipulating an independent variable (receiving the treatment or not) but, crucially, lacks random assignment of test subjects to treatment and control groups. When designed and executed well, quasi-experiments can produce results that are just as credible as some randomized trials.
When quasi-experiments are not feasible, make an educated guess. A best-guess counterfactual is a conjecture about what percentage of the estimated change in an outcome can be attributed to your program. For example, say the six-year college graduation rate among beneficiaries of the college prep program is 65 percent. Beneficiaries are evenly split by gender and are mostly Hispanic and black. The average six-year college graduation rate among Hispanic and black students in the U.S. is about 47 percent. Imagine the nonprofit served 100 students six years ago. Today, 65 of those students have graduated. Had those 100 students not participated in the college prep program, an estimated 47 of the 65 graduates would still have graduated, assuming that beneficiaries would have performed as well as their national peers. The nonprofit is therefore responsible for 18 graduates who otherwise would not have completed college.
Best guesses can be grounded in population statistics (as in the example above) or academic literature. Or, simply conduct the following thought experiment: Given the resources they have available, what would beneficiaries have done if your program did not exist? Would some have sought similar services from another organization? Would some have learned the skills that you teach through some other means? Would some have gained access to your life-saving product through their friends and family?
Tip #7. Cost: the other half of the story
In a world with limited resources and unlimited need, the amount of good that a program accomplishes is only half the story. Impact is the change in outcomes caused by a program relative to program cost. By choosing high-impact programs, we make our limited resources stretch further.
When calculating costs, it’s easy to get mired in line items. Use this organizing principle to ground your analysis: parallelism of costs and outcomes. Include all costs — but only those costs — that were necessary in generating the outcomes you report. Costs to train guidance counselors and tutors, subsidies for standardized testing fees, teaching materials and outreach costs to recruit students for the program — these costs are all directly related to improving the outcomes of beneficiaries of the college prep program. Fundraising costs, administrative costs, leadership team salaries — these costs are not directly implicated in beneficiaries’ outcomes and should not be included.3
Ready to report impact?
Social impact and its nuances may sometimes, at first glance, appear to resist quantification. But it’s far easier than many nonprofits think and we’re here to help. We hope you’ll put these tips to use with the Calculating Impact Series (step-by-step guides with a companion calculations template) — and in doing so, gain a vital tool to shout from the rooftops the real change you make possible for those you serve.
1 Less common, a program may explicitly seek to achieve multiple outcomes independent of one another. Imagine that the college prep program takes place immediately after school in order to prevent high schoolers from engaging in risky behaviors during the unsupervised period between the school bell and the return of their parents or guardians. The program has dual, independent aims: to lower youth delinquency and, by boosting college graduation, raise future incomes.
2 Besides bringing beneficiaries’ stories to life, anecdotes and case studies can be useful in understanding the nuances of a theory of change, such as the specific mechanisms that motivated beneficiaries to behave a certain way, what drove anomalous outcomes among certain beneficiaries, and the unintended (positive or negative) effects a program might have had on beneficiaries and non-beneficiaries.
3 Nonprofits often, by necessity, allocate non-program costs to grant budgets. These costs should not be counted in an impact calculation.