ImpactMatters measures the success of an emergency shelter as its cost to provide a night of shelter to people experiencing homelessness. Using “a night of shelter” as the metric of analysis is second best because it takes no account of differences in the quality of shelters. But “a night of shelter” does capture an important benefit and, important, is captured in existing datasets (unlike quality-focused outcomes). As datasets improve over time, more sophisticated measures of outcomes will emerge. To that end, we plan to research:
Number of beneficiaries who exited to “positive destinations,” as defined by the U.S. Department of Housing and Urban Development (HUD);
Beneficiary feedback on individual emergency shelters;
Measures of safety at individual emergency shelters;
The effect of co-locating other social services with emergency shelters.
We welcome your feedback on how we can better represent the work of emergency shelters in our methodology and on what resources we could provide to help nonprofits report the necessary data.
ImpactMatters generates estimates of impact — estimates that quantify the causal effect nonprofits have on social outcomes relative to cost. For example: a nonprofit shelters a person experiencing homelessness for an average of $50 a night. Our estimates incorporate best principles in social science,described in our Impact Methodology.
This document describes our methodology for estimating the cost-effectiveness of emergency shelters operated by nonprofits. HUD defines emergency shelter as “a facility with the primary purpose of providing temporary shelter for homeless people.”1 We distinguish among emergency shelters and five other categories of shelter recognized by HUD: transitional housing;2 permanent supportive housing;3 rapid rehousing;4 safe havens;5 and other permanent housing.6 The following methodology applies only to emergency shelters.
In this document, we describe emergency shelters as an intervention; the outcome by which we measure their impact; our methodology for estimating the cost-effectiveness of emergency shelters; and a checklist of data required from nonprofits to calculate cost-effectiveness.
Emergency shelters provide people experiencing homelessness with a temporary place to stay, typically either in a facility operated by the nonprofit or in a motel or hotel that accepts vouchers provided by the nonprofit. Beneficiaries can enter shelters in different ways depending on how the shelter network in a particular area is organized.7 In New York City, for example, a centralized intake center8 assesses beneficiaries’ needs and then places them in housing programs throughout the city. Meanwhile, the City of Los Angeles has a “no wrong door” model, whereby beneficiaries can drop in at any participating provider of social services and be referred to an appropriate shelter program.9 In most U.S. states, people in need can also dial 2-1-1 to speak with a community resource specialist for referrals to a shelter in their area.10
Many emergency shelters provide beneficiaries with a package of basic services in addition to a bed to sleep in: showers; toiletries; clothing; and laundry facilities. We consider these services part of “a night of shelter” because they are closely associated with having a residence. However, some emergency shelters provide extra services like meals, case management, job training and mental health services. To increase comparability across shelters, we exclude from our cost-effectiveness estimates the value as well as the cost of these extra services. See Nonprofit costs below for more.
The national median length of stay at an emergency shelter was 27 nights in 2017.11 After their stay, beneficiaries may exit to any destination, including another emergency shelter, a permanent housing facility or their own rented or owned permanent housing — or they may return back to street homelessness. In 2017, about 41 percent of all recorded beneficiaries across the country exited to a permanent housing destination after their stay in an emergency shelter, transitional housing program, safe haven or rapid rehousing program.12
We measure the success of emergency shelters as their cost to provide a night of shelter to a person experiencing homelessness. A night of shelter is what we would ordinarily call an “output” — a measure of the completion of activities designed to help beneficiaries achieve outcomes. We choose to estimate cost-effectiveness using the cost of a night of shelter for three reasons.
First, most emergency shelters are only one part of a “Continuum of Care” (C.o.C.) for people experiencing homelessness.13 According to HUD, each C.o.C. has four necessary parts: (1) outreach, intake and assessment; (2) emergency shelter as an immediate and safe alternative to sleeping on the streets; (3) transitional housing with supportive services to enable beneficiaries to develop skills for living in permanent housing; and finally, (4) permanent housing.14 Emergency shelters, considered in isolation from the rest of the continuum, are not intended to be a lasting solution to homelessness and its attendant effects on long-term outcomes like health and employment. Rather, emergency shelters are designed to offer beneficiaries immediate relief: a night of safe shelter.
Second, many emergency shelters report the number of nights of shelter they provided each year. For those that do not, the number of shelter-nights can often be inferred from data made available by HUD. This allows us to reach reliable conclusions for a large number of emergency shelters.
Finally, a night of shelter, though more of an output than a final outcome, is in and of itself a useful indicator of philanthropic success. The individuals and families that are housed in shelter reap an important benefit, as the nonprofit (and its donors) intend. In that respect, a shelter night is more useful as a metric of success than other outputs — for example, graduates from job training programs that may or may not serve the needs of trainees.
Methodology for estimating attributable outcomes¶
We estimate the causal effect of an emergency shelter on the outcome defined above: a night of shelter provided to a person experiencing homelessness. To do so, we count up the number of nights of shelter that the nonprofit reported providing over a specific period. For a detailed explanation of data sources and protocols for data interpretation, see Reference Manual on Data Analysis.
Then, we apply three assumptions:
The beds provided by neighboring emergency shelters are equivalent; we assume no important quality differences among emergency shelters. We assume all emergency shelters at least satisfy HUD’s minimum standards for safety, sanitation and privacy.15 See limitations below for how we address this oversimplification of the outcome.
The provision of shelter by Shelter A does not diminish the nights of shelter provided by any other (neighboring) emergency shelters — a plausible assumption for redistributive programs (“transfer programs”). This “counterfactual” assumption about the number of shelter nights provided in the absence of the program under review implies that the benefit of a night of shelter for a beneficiary in need constitutes a net gain; the gain is not offset by reductions in nights of shelter provided to other beneficiaries in need.
We assume there is, in general, no slack capacity in the homeless shelter system. In the absence of a given shelter, beneficiaries would not be able to stay at another shelter because other shelters are assumed to have no beds to spare. This assumption is based on national statistics showing that there were 0.7 beds available to serve each homeless person on average over the last five years.16
In algebra, we calculate the attributable outcomes of an emergency shelter (O) as follows:
Methodology for estimating cost¶
Below, we summarize the most important aspects of our methodology for estimating the costs of emergency shelters. For a detailed discussion of what sources of data we use, how we treat specific line items and accommodate variation in accounting practices, see Reference Manual on Data Analysis.
Costs we include¶
ImpactMatters estimates cost-effectiveness from the perspective of a socially minded donor. This means we count all important costs associated with a program regardless of who incurs them. Generally, the key cost-bearing parties are: the nonprofit itself; organizations with which it partners to run a program; the government (taxpayers); and the nonprofit’s beneficiaries.
We include in our calculation only those costs related to providing a night’s stay — and associated basic services (like showers and clean clothes) — at an emergency shelter.
Extra services for shelter residents¶
Besides a night of shelter, some nonprofits also provide meals and social services like case management and employment assistance. To achieve some measure of comparability across shelters, we exclude from our estimate of cost-effectiveness the value as well as the cost of ancillary services. If the nonprofit has not broken out the costs of extra services from total program costs but is evidently providing these extra services, we apply a standard cost adjustment (see Reference Manual on Data Analysis for details). Closely defining what services do and do not constitute a night of shelter allows us to analyze diverse programs fairly.
Other shelter and housing programs¶
Some nonprofits run an emergency shelter as well as other shelter or housing programs, such as transitional housing or permanent supportive housing. Wherever possible, we exclude from our calculation the value of these non-emergency shelter services as well as the costs of providing them.
If the nonprofit has not separated out programmatic costs in this way, we apply a standard cost adjustment. The cost adjustment is calculated using HUD’s Housing Inventory Count dataset.17 The Housing Inventory Count dataset reports the number of individuals sheltered by each nonprofit on a single night in January, broken out by six types of shelter and housing programs, including emergency shelter. This allows us to calculate the number of individuals sheltered as part of a nonprofit’s emergency shelter program as a percentage of the total individuals it sheltered across all programs. We then multiply the proportion by total programmatic costs, yielding an estimate of costs associated only with the nonprofit’s emergency shelter program. See Reference Manual on Data Analysis for more details on this calculation.
Other programs serving non-residents of the shelter¶
In addition to running an emergency shelter, a nonprofit may run other programs serving different beneficiary populations (i.e., non-residents of the shelter) — for instance, a workforce development program for low-income adults in general. This includes aftercare programs that serve former shelter residents as well as day shelter shelter services for non-overnight guests. We exclude from our calculation the costs of such unrelated programs as reported by the nonprofit. If the nonprofit has not broken out its programmatic costs by program, we are unable to complete a cost-effectiveness estimate for its emergency shelter program.
Many emergency shelters rely on government funding, particularly from HUD’s Emergency Solutions Grant.18 We assume that government costs are reflected entirely in the nonprofit’s expenses. In other words, we assume the government does not expend funds to provide emergency shelter beyond those passed through to each nonprofit.
After an initial period of free accommodation, some emergency shelters begin to charge beneficiaries an “overnight fee” for each additional night spent at the shelter. The fees are a cost to beneficiaries but an equivalent revenue for the nonprofit; the result is a net social cost of zero. We therefore need not make any adjustments to cost. We assume beneficiaries incur no other costs to stay at a shelter.
Methodology for calculating impact¶
To calculate the impact of an emergency shelter, we divide the total program-related costs incurred by all cost-bearing parties by the total nights of shelter provided. Crucially, the numerator and denominator must match logically: The numerator reflects the costs incurred in generating the attributable outcomes reflected by the denominator.
We determine whether an emergency shelter is cost-effective by asking whether there is a clear better alternative use of resources to achieve the same end. Specifically, we consider the following thought exercise: A nonprofit could spend programmatic resources to provide a night of shelter to a beneficiary or, alternatively, that beneficiary could purchase a night of shelter on the market. The nonprofit’s program is therefore cost-effective if its cost to provide a night of shelter is lower than the price a beneficiary would have to pay to obtain shelter on the market.
Our cost-effectiveness benchmarks are local Fair Market Rents from HUD’s dataset of small-area Fair Market Rents.19 The dataset provides Fair Market Rents by county and dwelling size (number of bedrooms). We use three-bedroom and one-bedroom Fair Market Rents, then convert monthly rental rates into daily rates. If the nonprofit reports serving individuals, we use the Fair Market Rent of a one-bedroom apartment in the county where the nonprofit operates. If it serves families, we use the Fair Market Rent of a three-bedroom apartment and divide it by the average household size — 3.5 — to estimate rent per individual. If some of a nonprofit’s shelters serve individuals and others serve families, then we calculate a weighted average of the two methods, where the weights are the estimated number of residents staying in individual versus family shelters.
Nonprofits that provide a night of shelter for more than 400 percent of Fair Market Rent are assigned 3 stars; for between 400 percent and 200 percent, 4 stars; and for less than 200 percent, 5 stars. We set the 4-star benchmark at 400 percent rather than 100 percent for a few reasons. First, as HUD acknowledges, “In almost all cases, the costs associated with providing housing for individuals and families who are homeless within a program exceeds the Fair Market Rent cost of providing rental assistance without supportive services.”20 While we have attempted to make corrections for these supportive services, these corrections may not always be adequate. Additionally, Fair Market Rents generally apply to long-term leases. Therefore, emergency shelters are in part solving a liquidity problem: Beneficiaries do not have the cash upfront to enter into a long-term lease.
Nonprofit checklist of data needed to calculate impact¶
The following data is necessary to estimate the impact of emergency shelters.
Required from nonprofit?
A program is a set of goods or services provided by the nonprofit to a population of beneficiaries with the goal of improving one or more outcomes. Generally, a program consists of the same components delivered to each beneficiary, with only minor deviations across different settings.
We recommend reporting whether beneficiaries of the emergency shelter also receive extra services not directly associated with a night of shelter (e.g., case management; job services; medical services; housing services; mental health services; legal services including court advocacy).
This refers to the program’s area of operation rather than where the nonprofit’s headquarters are located, if the two are different. We recommend specifying the program’s geography at the county level, which allows us to make any necessary adjustments for cost of housing across geographies.
If a nonprofit serves a special subpopulation of the homeless, we recommend that it report basic descriptors (e.g., age, gender, veteran status). For consistency, nonprofits may wish to use HUD’s standard descriptors.21
We recommend nonprofits report annual figures that align with their fiscal year.
Number of nights of shelter
We recommend nonprofits report the total nights of emergency shelter provided in a given period. If a nonprofit has not reported this, we calculate total shelter nights based on each nonprofit’s point-in-time count of sheltered individuals made available by HUD. See Reference Manual on Data Analysis for more.
Total program costs related to providing a night of shelter
We recommend nonprofits itemize or otherwise separate costs by those directly associated with a night of shelter and those associated with other social services. For nonprofits that run multiple shelter and housing programs, we recommend costs be broken out by type of shelter and housing program (e.g., emergency shelter costs separated from permanent supportive housing costs).
We recommend reporting beneficiary costs if they are substantial. Often, beneficiary costs are in the form of overnight fees, reported as program revenue. Beneficiary costs can be estimated at $0 if no fees are charged.
We recommend reporting partner costs if they are substantial. They can be estimated at $0 if they are not substantial. Partner costs are infrequently reported and are defaulted to $0 for emergency shelters.
Limitations of our analysis¶
Oversimplification of the outcome¶
Our standard outcome for emergency shelter is a night of shelter provided to someone experiencing homelessness. But there can be substantial differences in quality of shelters. Some offer residents individual rooms, some allow residents to stay for months, some have eligibility requirements. Additionally, a higher quality shelter may result in more rapid exit from the shelter system, and subsequently fewer nights spent there. As a result, our cost-effectiveness estimates capture the higher costs associated with better shelter but not the corresponding benefit to beneficiaries, such as greater relief and safety. When this happens, we provide commentary alongside the nonprofit’s cost-effectiveness estimate and star rating. The commentary conveys the ways in which our standard outcome measure oversimplifies the nonprofit’s achievements and whether we think our rating understates the nonprofit’s impact as a result.
Multiple important outcomes¶
We estimate the cost-effectiveness of a nonprofit’s program on a single outcome that best aligns with its stated mission. In the case of emergency shelters, that outcome is nights of shelter provided. There are other important outcomes associated with emergency shelter. For example, in cold months, a night of shelter can save a life. These additional outcomes are not distinct from shelter-nights, as is the case with food banks’ dual outcomes of meal provision and waste reduction. Instead, the primary outcome — a night of shelter — causes the additional outcomes. Accordingly, the night of shelter metric should capture these downstream outcomes.
Other distinct but important outcomes often associated with shelters, such as additional income generated from a shelter’s job training services, result from the supportive services that shelters offer, not the sheltering itself. Because we exclude the costs and outcomes of those services from our analysis, we do not analyze those additional outcomes.
Cost of reaching special populations¶
Often the people most in need are the most difficult — and costly — to reach. For emergency shelters, HUD identifies populations of this type including: victims of abuse, runaway youth, veterans and those with H.I.V./AIDS.22 Many nonprofits and donors are willing to spend more to reach these special populations. This may lead to a lower cost-effectiveness estimate relative to others. We typically address this by adjusting the counterfactual to reflect smaller likelihood of these beneficiaries otherwise receiving assistance. However, we already assume a zero counterfactual for all emergency shelters. Therefore, we do not adjust our ratings, though we specify on the nonprofit’s page that it is serving these populations.
Cost of working in special environments¶
Sudden, localized shocks like a natural disaster can dramatically affect the number of people seeking shelter. This should increase the nights of shelter provided by a nonprofit. This may also increase the cost of providing shelter, but since changes in shelter costs typically occur on a longer time-frame, we do not expect costs to rise disproportionately. Accordingly, we do not adjust our rating on this basis.
Cost of reaching special locations¶
Real estate is priced and taxed differently in different parts of the country, making it more or less costly to run an identical emergency shelter program depending on where the nonprofit chooses to work. We estimate county-by-county cost-effectiveness benchmarks for emergency shelters by adjusting local Fair Market Rents by the household size of those served by the shelter. This means comparing the cost-effectiveness estimate of an emergency shelter in Clark County, Indiana, to a different benchmark than that used for an emergency shelter in New York City.
Our estimates rely on data made public by nonprofits on their websites, annual reports, financial statements and Form 990s. There are, of course, ambiguities in the data and our interpretation of the data may not always match the nonprofit’s intention. For instance, we sometimes need to use context clues within descriptive fields to determine whether the nonprofit’s additional services are offered to non-residents or exclusively to residents. If ambiguities in the data are too large to be resolved with reasonable assumptions, we will not generate a cost-effectiveness estimate for that nonprofit.
For more detail on our sources of data and how we interpret them, please see Reference Manual on Data Analysis.
Representativeness of (analyzed) programs¶
We only issue ratings for nonprofits if we can perform analysis on 15 percent or more of the nonprofit’s total program budget. This approach means some nonprofits are rated on only some of their programs. The remaining programs, which we could not analyze, could be more or less cost-effective than the programs we analyzed.
Appendix: Common types of emergency shelters¶
Our methodology applies to programs where shelter is a central rather than peripheral activity. We would not, for instance, apply this methodology to a residential rehabilitation program for people facing substance abuse issues, but who would not otherwise be homeless.
In addition, our methodology applies to emergency shelters that provide a bed to sleep in. This may include cots, mats or “overflow” beds, which are available on an ad hoc basis when demand exceeds planned bed capacity. Our methodology does not apply to day shelters, which are open only during the day to provide homeless and low-income people with services like meals, showers and case management. We also exclude from our analysis drop-in centers, warming stations and cooling stations. While such facilities sometimes provide overnight shelter, their primary purpose is to provide other forms of immediate relief — relief from hunger or extreme weather, for instance. As such, HUD does not include these facilities in its annual point-in-time count of sheltered homeless people, but in its count of unsheltered homeless people.23
Below we list the common types of emergency shelters that are included in our analysis.
These programs provide beds (including cots or mats) in residential facilities.24
These programs provide vouchers or other forms of payment for homeless people to stay at a hotel or motel.
Programs classified by HUD as “other” provide beds to the homeless, but the beds are located in a church or other facility not dedicated for use by the homeless.
Programs serving specific populations¶
Some emergency shelters serve a specific population, which may be defined by age, gender and household type (e.g., families with children) or specific needs (e.g., runaway youth, people living with H.I.V./AIDS, veterans). Such shelters often have so-called “dedicated beds,” which must be filled by a person in a given subpopulation category.
Seasonal bed programs¶
Some emergency shelters increase their supply of beds or are only open during seasons when the demand for shelter is highest — usually in winter. Our methodology for counting nights of shelter is different for year-round programs than for seasonal programs, as explained in Reference Manual on Data Analysis.
Transitional housing provides people experiencing homelessness a place to stay combined with supportive services for up to 24 months.
Permanent supportive housing provides housing assistance and supportive services on a long-term basis to formerly homeless people. Permanent supportive housing projects funded by HUD’s Continuum of Care program exclusively serve beneficiaries who have a disability.
Rapid rehousing provides temporary housing assistance to people experiencing homelessness, moving them quickly out of homelessness and into permanent housing.
Safe havens provide temporary shelter and services to hard-to-serve individuals.
Other permanent housing is housing with or without services specifically for formerly homeless people, but that does not require beneficiaries to have a disability (by contrast to permanent supportive housing, which does).
Prevention Assistance and Temporary Housing office, or PATH, run by the N.Y.C. Department of Homeless Services
No Wrong Door resource by the City of Los Angeles
HUD: National Summary of Homeless System Performance 2015–17. Note that data on beneficiaries’ exit destinations are imperfect; beneficiaries sometimes leave unannounced or they either don’t provide or don’t have contact information for following up.
HUD defines a C.o.C. as a regional or local planning body responsible for coordinating the full range of homelessness services in a geographic area, which may cover a city, county, metropolitan area or an entire state.
Based on ImpactMatters analysis of annual HUD data on the size of the homeless population and inventory of beds available to serve that population. To calculate the number of beds available per person each year from 2014 to 2018, we divided the total number of beds available that year in emergency shelters, transitional housing and safe havens by the total size of the sheltered and unsheltered homeless population as measured by HUD point-in-time (PIT) count that year. We then averaged across the five-year period.
See HUD: PIT and HIC Data Since 2007. Any organizations that provide shelter and housing to people experiencing homelessness are required by HUD to report the number of individuals sheltered by their respective programs each year on a single night in January, regardless of whether a program receives government funding.
Per our Impact Rating Standard, we rate nonprofits that receive at least 15 percent of their revenue from donor contributions (as opposed to other sources of revenue, such as government funding, program-related earned revenue and investment income). Many emergency shelters are not rated because they rely exclusively or heavily on government funding.
HUD’s standard descriptors include 11 demographic subtypes (e.g., single males 18 years old and over, youth females under 25 years old) and two additional special populations (survivors of domestic violence, persons with H.I.V./AIDS). We also apply a standard “Veterans” descriptor to any emergency shelters that have dedicated beds for veterans.
U.S. Department of Housing and Urban Development: Notice for Housing Inventory Count (HIC) and Point-in-Time (PIT) Data Collection for Continuum of Care (CoC) Program and the Emergency Solutions Grants (ESG) Program