Our 3-, 4- and 5-star ratings are awarded based on cost-effectiveness (a nonprofit’s impact relative to a benchmark). To issue these ratings, we first calculate the impact of a nonprofit’s program (attributable outcomes relative to cost). We then assign ratings based on the question: is this a good use of resources by that nonprofit, given viable alternatives?
Cost-effective programs are making good use of resources, given the alternatives. Nonprofits may be not cost-effective because they are implementing interventions that don’t work or implementing them in ways that don’t work. One way of answering the question “Is this cost-effective?”: consider whether there is a clear better alternative. For example, we set our threshold for soup kitchens based on the cost of meals to low income people in that city. If the cost to a soup kitchen to deliver a meal is substantially higher, we would be better off using those resources elsewhere. This is a subjective exercise and we welcome input. Often, the threshold varies by geography or beneficiary type.
If the nonprofit meets the threshold that has been set for its type of program, it receives five stars.
A simplified example: let’s assume that $3 is the average cost of a meal in a particular city.
Soup Kitchen A delivers a meal for $1. Since that is substantially lower than $3, it earns 5 stars.
Soup Kitchen B delivers a meal for $4. Since that is close to $3, it earns 4 stars.
Soup Kitchen C delivers a meal for $6. Since that is much larger than $3, it earns 3 stars.
The four- and five-star thresholds are determined independently, so it is possible that a nonprofit may be less effective than its peers but still more effective than the threshold. In this case, we award five stars.
Some nonprofits may be cost-effective, but the effects may not be that important. For example, a program that boosts income by $5 per year is not having an important effect; one that is boosting income by $500 is. We take this into account, using research and expert opinion to identify what levels of impact represent an important difference in the lives of beneficiaries.
The standard process described above applies to most ratings we issue. But the social sector is complex and programs can be diverse in their design, setting and other aspects that affect their cost-effectiveness. Wherever possible and appropriate, we try to capture that complexity by adjusting our estimates and ratings, explained in detail here. For example, we may adjust ratings upward if a nonprofit has adequate evidence that its program generates other important outcomes not captured in the impact estimate. If we lack the data to make quantitative adjustments, we provide commentary alongside our impact estimates. Our objective in doing so is to give credit where it is due.