You are walking down the street to get to an appointment a mile away. Walking there should take you 20 minutes. A friend drives by, sees you, and gives you a lift. Instead of taking 20 minutes, you now get there in 5 minutes. Would you have gotten to your destination even if you didn’t get the ride? Chances are, yes. If you were to measure the impact of your friend's intervention in your journey, they wouldn't get credit for getting you to your destination (since you would have gotten there without them). But — your friend would deserve credit for saving you 15 minutes.
Counterfactual analysis is the difference between what did happen to people served by a nonprofit and what would have happened to them without the nonprofit. It’s not just an academic exercise. Take fictional Nonprofit A, which provided coaching and counseling to job seekers. Of the 100 people it served last month, 50 got jobs. Is that A’s impact? The answer is almost certainly “no.” The people coming through the door already wanted jobs — odds are they were job hunting in other ways. So there is a good chance that at least some of them would have gotten a job without the nonprofit’s help. Therefore, we shouldn’t count all 50 jobs as a success for the nonprofit. After all, the nonprofit is trying to get people jobs who otherwise wouldn’t have them, and some of those 50 would have had them.
How many jobs can Nonprofit A take credit for? That’s the core of counterfactual analysis.
For a more technical discussion of our methodology, see here.
The funny thing about the counterfactual — you can never truly know it. The very nature of the counterfactual is that it is counter to fact. Unless we can conjure up an alternate universe, we can't observe what would have happened if you had continued to walk because, in fact, you took the ride. However, we can make a good attempt at estimating the counterfactual. Broadly speaking, we can do three things. First, we can run a research study called a randomized trial. Second, we can run quasi-experimental studies. Third, we can look at the studies done by others in the past and use those to come up with an estimate.
How it works
Let’s say a nonprofit has enough money to serve 100. Rather than identifying 100 people, it identifies 200 and randomly assigns them to get the program or not get the program. When we split a large enough population in two by random assignment, the characteristics of both groups are roughly the same. Try randomly splitting a jumbo-sized bag of m&ms in two — in each group, there will be about the same number of red, green, blue and so on. The nonprofit then delivers the program to one group (the treatment group) and tracks how it fares over time. The nonprofit also tracks how the other group (the control group) fares over time. If the treatment group does better (for instance, they earn more) relative to the control group, we know the only thing that could explain the difference is the nonprofit’s program. If the two groups do the same, we conclude the program didn’t work.
Large programs, if feasible, should generally go through a randomized trial at some point, especially if they involve a novel program model or are bringing an existing model to a new and different population. It may not be appropriate for smaller programs, but it shouldn’t be ruled out.
How it works
Often nonprofits lack the ability to conduct a randomized trial. In those situations, the nonprofit can use other methods to estimate the counterfactual. For example, consider a scholarship given to people who score 1200 out of 1600 or higher on their SATs. We want to know if the scholarship increases college graduation. We could compare people who got the scholarship to those who didn’t. But it’s likely that people with higher SAT scores — like those who received the scholarship — graduate at higher rates than those who don’t. Simply comparing those who received scholarships (higher SATs) to those who didn’t (lower SATs) will make the program look better than it is. A better option is to look at people who scored right around the cutoff for the scholarship — say 1150 to 1250. They are all great students and the differences in their ability are probably pretty trivial. The people right above the cutoff (1200-1250) received the scholarship, but are otherwise similar to people right below the cutoff (1150-1200). The two groups are therefore comparable. We can conclude that if the higher group did better than the lower group, it was likely because they received the scholarship and not because they were much better students over all.
Some basic quasi-experimental techniques are accessible to many nonprofits, but often expert help is needed.
How it works
For many programs, the answer is to turn to past studies of similar interventions. For example, many studies have examined which job training programs work. One particular job training program, supported employment, has been shown to increase the number of people with disabilities who get jobs. Rather than study each program individually, we can take a measure of success produced by the nonprofit — in this case, the number of people who got jobs — and subtract from it an estimate of the counterfactual borrowed from an academic study. The result is the number of people who got jobs as a result of going through the program.
Using research literature is a cheap and fast way to estimate the impact of a nonprofit. The nonprofit must still collect some data — such as an output or a proxy for an outcome — but it doesn’t need to participate in a study.
For many nonprofits, using academic literature is the best option.
At ImpactMatters, we are obsessed with the counterfactual. Why? Anything short of counterfactual analysis would produce incorrect estimates of impact. By taking due account of the counterfactual, we can be confident in identifying top nonprofits — and then sharing those nonprofits with you.
If you really want to get in the weeds, check out our Impact Rating Standard.