Postsecondary Scholarships

Overview

ImpactMatters generates estimates of impact — estimates that quantify the causal effect nonprofits have on social outcomes relative to cost. For example: for $10,000, a nonprofit increases the income of a scholarship recipient by $16,000. Our estimates incorporate best principles in social science, described in our Impact Methodology.

This document describes our methodology for estimating the cost-effectiveness of scholarship programs run by nonprofits. In addition, we describe common reasons for variation in cost-effectiveness and provide a checklist of data required from nonprofits to calculate cost-effectiveness.1


Description

Education can be a powerful tool for economic mobility in the United States. College graduates achieve higher earnings, experience lower rates of unemployment and enjoy better health. Yet the benefits of a college education are not equally distributed; students from high income households are significantly more likely to enroll in college compared to their low income peers.2 And as the costs of higher education continue to rise,3 students turn to loans to make ends meet. Student loans now make up the largest proportion of U.S. non-housing debt.4 Borrowing against future income can be daunting and is not possible for some. As a result, college education is a prohibitively expensive endeavor for some students.

Scholarship programs aim to address this challenge by helping students in need meet their educational and living expenses while in school.5 The majority of scholarship programs support students pursuing postsecondary coursework or a degree,6 although some programs support high school students. We focus exclusively on postsecondary scholarships given the preponderance of focus by nonprofits on postsecondary rather than high school scholarships. These programs often target specific groups of individuals such as low income students or students from a specific racial or ethnic background. For a comprehensive list of types of scholarship programs, see the Appendix.


Outcome

Outcomes: Increase in postsecondary enrollment, persistence and graduation; reduction in debt7
Final outcome: Increase in income (lifetime earnings and value of scholarship) for a scholarship recipient

A scholarship in theory improves the outcomes of recipients through two potential paths. First, by reducing the cost of education, the scholarship may increase matriculation, learning, graduation and then finally lifetime earnings. The social science literature has consistently found evidence that scholarships increase the likelihood of matriculation to college.8 There is also evidence that scholarships boost college graduation rates, although the magnitude of this effect varies considerably by type of school (community college versus four-year), type of award (merit, need based or neither), recipient profile and size of the award. And wage data consistently demonstrate significant earnings differentials for adults with higher education levels compared to adults with lower education levels.

Second, a scholarship may reduce debt (or free up other resources). In addition to reducing financial constraints, student debt relief can increase borrowers’ geographic mobility and probability of changing jobs, and increase their income in the long run.9 There is also suggestive evidence of non-pecuniary returns to education including better health, happier marriages and more successful children.10 In our analysis of nonprofits that provide scholarships, the effect of both paths is combined and estimated as additional income.


Methodology for estimating attributable outcomes

We estimate the causal effect of a scholarship program on the outcome defined above: increase in income for a student in need. To do so, we rely on the research literature to estimate the effect of a scholarship on student graduation rates and expected average additional lifetime earnings. We then combine the estimated increase in lifetime earnings with the income boost students receive from the scholarship itself to estimate the expected total average increase in income for the scholarship recipient.

Specifically, we take the following steps:

Step 1: Calculate scholarship coverage ratio (average proportion of educational costs covered by the nonprofit’s scholarship). We divide the average dollar value of the scholarship (provided by the nonprofit) by the average price of postsecondary education. If the nonprofit provides the names of institutions where scholarship recipients are studying, we use the price of education at those institutions. Otherwise, we use the average price of education at institutions in the same state and of the same type (public or private) as specified by the nonprofit.

Step 2: Estimate the increase in number of graduates. Seven studies directly inform our estimates: Angrist et al (2016)11, Bartik et al (2019)12, Denning et al (2019)13, Anderson and Rab (2018)14, Anderson (2019)15, Fack and Grenet (2015)16, and Sjoquist & Winters (2015)17. Angrist et al. (2016) conducted a randomized evaluation to examine the effect of a generous, privately funded merit- and need-based scholarship program for students at public colleges in Nebraska. Bartik et al. (2019) used a regression discontinuity design to estimate the effect of the Kalamazoo Promise scholarship program, which is neither merit nor need based. Denning et al. (2019) used administrative data from Texas public colleges and a discontinuity in grant generosity to estimate the effects of additional Pell grant awards for college students.18 Similarly, Anderson and Rab (2018) used a randomized design to evaluate the effect of supplemental scholarship aid for Pell-eligible community college students in Wisconsin. And Anderson (2019) exploited an eligibility cutoff to study the effect of a need-based financial aid program on completion of associate’s and technical degrees. Fack and Grenet (2015) used comprehensive administrative data on France’s largest financial aid program and discontinuities in the grant eligibility formula to estimate the effect of scholarships on graduation rates for master’s degree students.19 Finally, Sjoquist and Winters (2015) reviewed data on the effectiveness of 25 state merit aid programs in the U.S.20

Together, these studies provide the best evidence on the causal effect of scholarships on college graduation rates. From these studies, we take nine graduation and enrollment treatment effect estimates for scholarships. These treatment effect estimates vary by key features along three dimensions: award criteria21 (merit based, need based, both or neither22), school type (vocational certificate, associate’s, bachelor’s or master’s degree), and coverage (percent of educational costs covered by the scholarship being studied).23 The effects are summarized in Table 1.

Exact Match: For scholarship programs that exactly match the “award criteria” and “school type” features of one of the nine estimates, we use the treatment effect from the exact match (“G” = unadjusted comparable treatment effect).

Fuzzy Match: For scholarship programs that do not exactly match the “award criteria” and “school type” features of one of the nine estimates, we identify the study most qualitatively similar to the nonprofit, and use the estimated treatment effect. With only nine estimates, we do not have enough data to quantitatively estimate the relative effects on graduation for each combination of the four award criteria and four school types. Table 1 summarizes how we fuzzy-match certain scholarship program types to treatment effects from the literature. As with Exact Match, the treatment effect from the fuzzy match is recorded as “G” = unadjusted comparable treatment effect.

For programs that offer a combination of scholarships (e.g., half of all scholarships are awarded to students pursuing associate’s degrees and half to students pursuing bachelor’s degrees), we take the average of the relevant treatment effects.24

If a nonprofit reports both the number of scholarship recipients who graduate and a quality counterfactual, then we use both these pieces of data. We subtract from the number of graduates the number expected to have graduated even if the scholarship had not, in the counterfactual scenario, been provided. We then adjust for scholarship coverage (step 3 below) and estimate the increase in lifetime earnings (step 4). If a nonprofit reports outcomes without a counterfactual, then we estimate the counterfactual from the most closely matched study from the research literature.25

Table 1: Enrollment and Graduation Treatment Effects from the Literature

Source

Criteria

School Type

Coverage Ratio

Graduation Treatment Effect (G)

Enrollment Treatment Effect

Exact Match

Fuzzy Matches

Autor 2019 presentation of Angrist et al. (2016, new version pending) findings

Both

B

76.0%

8.50%

11.90%

B Both

Bartik et al. (2019)

Neither

B

82.5%

7.90

6.00%

B Neither

Denning et al. (2019)

Need

B

2.8%

3.30%

0.00%

B Neither

Fack & Grenet (2015)

Need

M

23.8%

4.90%

3.70%

M Need

M Neither; M Both

Autor 2019 presentation of Angrist et al (2016, new version pending) findings

Both

A

88.1%

3.80%26

5.60%27

A Both

VC Both

Bartik et al. (2019)

Neither

A/VC

82.5%

3.70%

0.00%

A Neither

VC Neither

Denning et al. (2019), Anderson & Rab (2018)

Need

A

23.9%28

0.37%29

1.86%

A Need

Anderson (2019)

Need

VC

7.2%

1.50%

0.00%

VC Need

Sjoquist & Winters (2015)30

Merit

A/VC/B/M

0%

0.00%

A/VC/B/M Merit

Note that in the above table:

  • VC = Vocational Certificate

  • A = Associate’s degree, including Associate of Arts (AA) and Associate of Science (AS)

  • B = Bachelor’s degree, including Bachelor of Arts (BA), Bachelor of Science (BSc), etc.

  • M = Master’s degree, including Master of Arts (MA), Master of Science (MS), etc.

Step 3: Adjust comparable treatment effect based on scholarship coverage ratio. From a simple regression of the treatment effects from five of the studies31 on the scholarship coverage ratio in each study, the slope on the scholarship coverage ratio is 0.0457. See Figure 1, below. We use this slope estimate to adjust “G” (unadjusted comparable treatment effect) by the difference between the scholarship coverage ratio for each nonprofit and the scholarship coverage ratio from the study chosen as most comparable to that nonprofit.

Specifically, this generates:

G’ = G + (SCRNP - SCRComp) * 0.0457
where:
G’ = Adjusted comparable treatment effect on likelihood of graduation (percentage points)
G = Unadjusted comparable treatment effect (from step 2 above)
SCRNP = the nonprofit’s scholarship coverage ratio
SCRComp = the scholarship coverage ratio from the study deemed most comparable
0.0457 = slope from correlation between scholarship coverage ratio and treatment effect across the six studies


Figure 1. Correlation between scholarship coverage ratio and treatment effect on graduation

Graphing the treatment effects from five studies relative to the scholarship coverage in each study (as shown in Figure 1) demonstrates a positive relationship between scholarship value and graduation effect. We assume linearity between scholarship coverage and treatment effect on graduation. In other words, the greater the proportion of educational costs covered by a nonprofit’s scholarship, the greater the scholarship’s effect on the recipient’s probability of graduating with a degree. Assuming linearity is likely an oversimplification but a necessary one given that the literature lacks any insight on the effect of comparable scholarships of different sizes on graduation effects.

Step 4: Estimate the increase in lifetime earnings. We estimate earnings by using the Current Population Survey (C.P.S.) data from the U.S. Census Bureau. These data give us median salaries by education level, broken down by race and gender. We take median earnings data by education level for people 25 years of age and older who had any work experience in the past year.

Because returns to education vary significantly based on socioeconomic and demographic characteristics,32 we adjust our estimates based on student characteristics when known. For example, if a nonprofit scholarship targets women, we apply the earnings differential between degree-holding and non-degree-holding women based on C.P.S. data. We also adjust earnings differentials when a scholarship targets low income students. Using analysis from Bartik and Hershbein (2018), we estimate that the returns to education for low income students are only 72 percent of what they are for non low income students.33 When a scholarship targets low income students, we scale back the C.P.S. earnings boost by 28 percent.34

C.P.S. data reports earnings for adults with high school degrees, associate’s, bachelor’s and master’s degrees but not vocational certificates. Instead, to estimate the earnings return to a vocational certificate, we rely on a literature review from Belfield & Bailey (2017). The authors estimate the average return to a vocational certificate is 8 percent and 22 percent higher relative to high school earnings for men and women, respectively. We apply these returns to the C.P.S. estimates of high school earnings to estimate the returns to vocational certificates.

Once we know the annual earnings boost that matches the demographic and income characteristics of the scholarship recipients, we multiply it by the assumed duration of recipients’ working lives (43 years between the ages of 22 and 65). We then apply a social discount rate and additionally account for inflation over time, translating the cumulative stream of future earnings into present value.35 For adult and non-traditional learners, we adjust the estimates based on estimated time to retirement.

Using median earnings data from C.P.S. requires two assumptions. First, that wage comparisons across education levels generally reflect the causal returns to education. We follow the precedent of other researchers (Levin et al. 200736 and Bartik et al. 201637, for example) in making this assumption.38 If this assumption is incorrect, it is at least possible that the distortion resulting from use of C.P.S. data happens at every level of education proportionally such that the difference between estimates could still be fairly accurate for our scholarship population even if the estimates themselves at each level are too low or too high. If anything, the comparison of Denning et al.’s estimates of the returns to a bachelor’s degree and our estimates with C.P.S. data suggest that observational data underestimate the differential returns to education.

Second, that the marginal student who attains a higher degree because of a scholarship will enjoy the same earnings increase as the average estimate. In their estimation of the benefits and costs of a place-based scholarship program, Bartik et al. (2016) make this same assumption.

Step 5: Estimate causal effect on student income, including the expected average boost in lifetime earnings and the increase in income enjoyed by scholarship recipients due to the scholarship itself. Critically, we only count scholarships as transfers of income to those students whose enrollment decisions are not affected by scholarship receipt.39 This is because the scholarship saves those students expenses they would otherwise have incurred.

In algebra, we calculate the attributable outcomes of a scholarship program (O) , measured in additional income, as follows:

O = (N * G’ * E) + S * (1 - R)
where:
N = Number of scholarship recipients (from nonprofit)
G’ = Percentage point increase in graduation rate caused by scholarship, from literature and adjusted for coverage (step 3)
E = Additional earnings for a graduate compared to non-graduate — cumulative over working life and discounted to present value. Non-graduate assumed to obtain one level of education lower.
S = Total dollar value of scholarships provided to all students (from nonprofit)
R = Percentage point increase in enrollment, from the literature.

Methodology for estimating cost

Below, we summarize the most important aspects of our methodology for estimating the costs of scholarship programs. For a detailed discussion of what sources of data we use, how we treat specific line items and accommodate variation in accounting practices, see Reference Manual on Data Analysis.

Costs we include

ImpactMatters estimates cost-effectiveness from the perspective of a socially minded donor. This means we count all important costs associated with a program regardless of who incurred them. Generally, the key cost-bearing parties are: the nonprofit itself; organizations with which it partners to run a program; the government (taxpayers); and the beneficiaries.

Nonprofit costs

Scholarship programs are largely cash operations. The majority of their revenues tend to be financial contributions and the majority of their expenses tend to be scholarships provided. Other program service expenses may include employee compensation, advertising and promotion, fundraising and related expenses incurred in the administration of the scholarship program. All direct, monetary costs incurred by the nonprofit to run its scholarship program are included in our calculation. Scholarship providing nonprofits generally report these expenses on the Form 990.

Some nonprofits disburse grants to educational institutions for academic programming and support, in addition to and separate from scholarship grants. Other scholarship providing nonprofits offer additional services like mentorship and career counseling for scholarship recipients. Wherever possible, we exclude from our calculation the costs associated with these non-scholarship activities, as reported by the nonprofit.

If the nonprofit has not separated out programmatic costs in this way, we apply a standard set of assumptions to isolate scholarship program costs. See Reference Manual on Data Analysis for more details of this calculation.

Beneficiary costs

The primary cost incurred by students is the cost of pursuing a postsecondary degree. In reality, some of this cost is covered by the nonprofit’s scholarship and the rest is covered by student and family resources, loans, work study and other grants. There is some evidence that private scholarships reduce student reliance on loans and work study. In the long run, this can lead to a reduction in student debt and resulting cost savings for the student. While reduction in student debt is an important downstream outcome, we lack sufficient information to estimate the resulting cost savings.

We only include the cost of a degree for those students who graduated because of the scholarship program.40 Some students, in the counterfactual case, would still have attended and perhaps graduated, so we will not count the societal costs associated with their degrees. This is a simplification. In reality, some societal resources would have been expended regardless of a student’s decision to pursue college (e.g., grants earmarked to be disbursed in a particular academic year, with no rollover), and so, shouldn’t included as a societal cost. Other societal resources, like the entitlements-like Pell grant program, are almost entirely responsive to demand. We assume all resources are expended based on demand.

Partner organization costs

Nonprofit scholarship programs that disburse financial awards directly to recipient students do not rely on any partner organizations, so we assume partner costs for these nonprofits to be zero. Nonprofit scholarship programs that disburse financial awards to the educational institutions where recipients are studying rely on these partner organizations for disbursal of the scholarship awards. Unless we have demonstrable evidence that the partner organization is incurring costs in disbursement of the award, we assume partner costs to be zero.

Government costs

Nonprofit scholarship programs do not usually partner with government agencies (at the local, state or federal level) for disbursement of scholarships. Instead, nonprofit scholarships are private awards to individual students and educational institutions where the students are enrolled. So there are no direct costs incurred by government partners.

Receipt of a private scholarship award may reduce a student’s reliance on federal loans, but scholarships also change students’ college choices, sometimes pushing students to enroll in more expensive colleges.41 So the net effect of a private scholarship on government loans (and government costs) is unclear. The effect is also likely minimal, given that the average scholarship amount for full-time undergraduates is about $3,800 whereas the U.S. Federal Student Aid office at the Department of Education provided more than $122.4 billion in federal grants, loans, and work-study funds to approximately 12.7 million students in fiscal year 2018.42 In other words, the effect of any single scholarship on U.S. financial aid expenses is likely minimal. In addition, we lack information on the effect of each private scholarship on student loans from government sources. For these reasons, we assume government costs to be zero.


Methodology for calculating impact

To calculate the impact of a scholarship program, we divide aggregate attributable outcomes (additional income for all scholars) by aggregate costs incurred by all cost-bearing parties. Crucially, the numerator and denominator must match logically: The denominator reflects the costs incurred in generating the attributable outcomes reflected by the numerator.


Cost-effectiveness benchmarks

To rate the cost-effectiveness of scholarship programs, we apply our standard benchmarks for programs that aim to boost the income of beneficiaries. We use the same set of benchmarks whether programs aim to boost income immediately via a transfer of resources or in the future by, for instance, raising beneficiaries’ earning potential, or both. The benchmarks are as follows:

  • 4 stars: Programs that boost income by 85 percent as much as total program cost

  • 5 stars: Programs that boost income by 150 percent as much as total program cost

Functionally, these benchmarks mean that a scholarship program must be generating close to as much in future income than it expends in costs to run its scholarship program in order to earn 4 stars and substantially more in future income to earn 5 stars.


Nonprofit checklist of data needed to calculate impact

The following data is necessary to estimate the impact of postsecondary scholarship programs.

Table 2

Checklist item

Required from nonprofit?

Details

Program activities

Yes

A program is a set of goods or services provided by the nonprofit to a population of beneficiaries with the goal of improving one or more outcomes. Generally, a program consists of the same components delivered to each beneficiary, with only minor deviations across different settings.

Geography

Yes

This refers to the program’s area of operation rather than where the nonprofit’s headquarters are located, if the two are different. We recommend specifying the program’s geography at the state level. This allows us to estimate average educational costs in the correct state (unless the nonprofit has provided the name of a specific educational institution, in which case we will use the educational costs for that institution).

Beneficiary type

Yes

If the nonprofit targets a specific beneficiary population besides students in general, we recommend providing basic descriptors of that population (e.g., gender, race, income level, school-age versus non-traditional students).

Timeframe

Yes

We recommend nonprofits report annual figures that align with their fiscal year.

Eligibility requirements

Yes

We recommend nonprofits report which of these eligibility requirements best describe their scholarships: (a) need based; (b) merit based; (c) both need and merit based; or (d) neither need nor merit based.

Destination institution

No

If a nonprofit provides scholarships for study at specific institutions, we recommend specifying the name of those institutions.

Type of institution

Yes

If a nonprofit does not limit scholarships to specific institutions, we recommend reporting whether scholarship recipients attend (a) public schools, (b) private schools, or (c) both.

Type of degree

Yes

We recommend nonprofits report the level of education pursued by their scholarship recipients: (a) associate’s degree; (b) bachelor’s degree; (c) master’s degree; (d) vocational certificate; or (e) some specific combination of the aforementioned options.

Number of scholarship recipients

Yes

We recommend nonprofits report the total number of students who received scholarship funding over the specified timeframe.

Total scholarship funding awarded

Yes

We recommend nonprofits report the total value of scholarships provided over the specified timeframe, excluding any grants awarded for purposes other than helping students cover the costs of tuition, required fees, room, board and living expenses.

Graduation rate among scholarship recipients

No

We recommend nonprofits report the average six-year graduation rate among their scholarship recipients. This may be the graduation rate for a recently graduated cohort of scholarship recipients or the graduation rate among all scholarship recipients to date. If this data is not available, we will fill it in with an estimate from the social science research literature.

Counterfactual graduation rate

No

This is not just the percent of scholarship recipients who graduated, but how many of those students would have graduated if not for the scholarship. If this data is not available, we will fill it in with an estimate from the social science research literature.

Program cost

Yes

We recommend reporting total program costs related to scholarship disbursement, including the actual amount of scholarships awarded but excluding any costs of non-scholarship activities. Non-scholarship activities could be programs for beneficiaries other than scholarship recipients (e.g., job training for formerly incarcerated adults) or extra services provided to scholarship recipients (e.g., mentoring and test preparation).

Beneficiary cost and partner cost

No

The balance of costs not covered by the nonprofit’s scholarship is likely covered by student and family resources, loans, work study and other grants. We only apply this cost to those students who graduated as a result of the scholarship program. If nonprofits do not track this data, we calculate an estimate based on the cost of education at the institution(s) and degree level(s) specified by the nonprofit.

Limitations of our analysis

Oversimplification of the outcome

Using “increased income” as our metric of analysis does not intuitively convey the potential downstream benefits of receiving a scholarship. For example, there may be psychic benefits like less stress associated with paying out of pocket for school, paying off loans or working while going to school. Additionally, our methodology may simplify the differing effects a scholarship has on different populations, such as refugees or people with illnesses.

Cost of reaching special populations

Some nonprofits may have to incur additional costs to reach particularly disadvantaged students. For instance, a nonprofit may have to spend more on outreach to make potential first-generation college students aware of its scholarship program.

Specific counterfactuals

To understand the impact of a program, we ask the counterfactual question: What would have happened to beneficiaries if the program had not, counter to fact, been there to serve them? Because the vast majority of nonprofits have not conducted impact evaluations, we need to construct our own counterfactuals based on public data sources and the research literature. But in doing so, we risk masking variation in effectiveness across nonprofits. For instance, under our methodology, a $5,000 need-based scholarship for associate’s degree students at state schools in Texas has the same effect on a student’s probability of graduation as a $5,000 need-based scholarship for associate’s degree students at private colleges in California. And a graduate with a bachelor’s degree in computer science who grew up in a moderately low-income household has the same future earnings as a graduate with a bachelor’s degree in social work who came from the lowest income bracket.

Data quality

Our estimates rely on data made public by nonprofits on their websites, annual reports, financial statements and Form 990s. There are, of course, ambiguities in the data and our interpretation of the data may not always match the nonprofit’s intention. For instance, a nonprofit might describe its scholarship recipients as being “in need” without specifying their income bracket. We interpret students “in need” as those coming from households with incomes 1.85 times the poverty level and base our calculations on the expected future earnings of students from this demographic. For more detail on our sources of data and how we interpret them, please see Reference Manual on Data Analysis.

Representativeness of (analyzed) programs

We only issue ratings for nonprofits if we can perform analysis on 15 percent or more of the nonprofit’s total program budget. This approach means some nonprofits are rated on only some of their programs. The remaining programs, which we could not analyze, could be more or less cost-effective than the programs we analyzed.


Appendix: Common types of scholarship programs

Our methodology applies to programs where scholarship provision is a central rather than peripheral activity. We would not, for instance, apply this methodology to a job training program for unemployed adults just because the program provides small scholarships for professional study. We only consider nonprofits for which the provision of scholarships accounts for at least 15 percent of programmatic activities.

Scholarship programs usually support individual students in pursuit of a high school degree, postsecondary43 coursework or a postsecondary degree by providing financial resources to help cover the costs of tuition and fees, room and board, books and supplies, personal costs such as transportation, and related educational expenses. These programs often target specific groups of individuals like low-income students or students from a specific racial or ethnic background.

Most scholarship programs in the U.S. share a common model: these nonprofits collect resources to fund scholarships, promote the scholarship opportunity, oversee the application process, identify applicants to receive scholarships, administer the awards, and follow up with recipients over time. While most scholarship programs follow this basic program model, they may differ in factors like processes, award type and timing, and beneficiary group targeted.

Purpose of Scholarship

The majority of scholarship programs fall into two categories:

Two-year degree programs: providing scholarships for students pursuing associate’s degrees and technical certificates

  • Community colleges offer academic courses for associate’s degree, technical courses/certificates, and continuing education courses

  • Junior colleges are similar to community colleges but usually private

  • Some institutions strictly offer technical/vocational programs

Four-year degree programs: providing scholarships for students pursuing a bachelor’s degree

  • Colleges grant bachelor degrees; some also award associate degrees and master’s degrees

  • Universities grant bachelor’s and master’s degrees

Note: The titles “two” and “four year degree programs” are more reflective of naming conventions for the type of scholarship program, rather than firm time limits. Scholarship programs for associate’s degrees are not generally strictly limited to two years, (or bachelor’s to four years), but rather usually tend to be tied to the length of time to completion for the given degree.

In addition to these categories, a minority of scholarship programs are:

  • High school programs: providing scholarships for students to attend private high schools

  • General educational support programs: providing support to students in pursuit of continuing education, technical or vocational education, or postsecondary coursework not leading to a degree

  • Master’s degree programs: providing scholarships to students to pursue master’s level graduate degrees

  • PhD programs: providing scholarships, and often living stipends, to PhD students

We focus exclusively on postsecondary scholarships (for vocational certificates, associate’s, bachelor’s and master’s degrees) given the preponderance of focus by nonprofits on postsecondary rather than high school or PhD scholarships.

Processes: Direct v. Indirect Disbursement

Many scholarship programs manage the application, review, and disbursement processes directly. These nonprofits manage their scholarship applications, review and identify awardees, and directly transfer scholarships to individual recipients. Alternatively, some nonprofits play an indirect role in scholarship disbursement by transferring scholarship awards to the educational institutions where individual recipients are studying. These indirect disbursement nonprofits may review applications and select individual scholarship recipients, or provide a lump sum to an affiliated educational institution to identify and administer scholarship awards to students.

Award Criteria

Scholarship criteria broadly fall into one of four buckets:

  1. Merit: Merit based programs have an academic achievement requirement at the time of application — usually high school grade point average (G.P.A.) or being in a top percentile of the graduating class. Many merit and need based programs also have college G.P.A. requirements for continued scholarship eligibility. We use academic performance prior to award receipt to distinguish merit programs

  2. Need: Need based programs have an eligibility requirement related to financial need.

  3. Both: Scholarship programs that require students to demonstrate both academic achievement and financial need at the time of application.

  4. Open eligibility/neither: The fourth category includes scholarships that are neither merit nor need based and instead target a broad group of students (e.g., all high school graduates from a school district). These are sometimes referred to as place-based scholarship programs.

Target Beneficiary Group

Scholarship programs often target specific beneficiary groups. These may be rather broad, such as need based scholarships that target low income students, or highly specific, such as women who intend to pursue a career in law in Texas, or scholarships reserved for the children of members of a union.




Footnotes

1

This methodology was developed in collaboration with Dean Karlan, Shannon Coyne and Jonathan Vayness at Northwestern University.                                                                                                                      

2

J-PAL: Navigating the rocky road to higher education <https://www.povertyactionlab.org/blog/9-10-18/navigating-rocky-road-higher-education>`_                                                                                                                      

3

According to the U.S. Department of Education National Center for Educational Statistics, between 2005–06 and 2015–16, prices for undergraduate tuition, fees, room, and board at public institutions rose 34 percent, and prices at private nonprofit institutions rose 26 percent, after adjustment for inflation. For the 2015–16 academic year, annual current dollar prices for undergraduate tuition, fees, room, and board were estimated to be $16,757 at public institutions, $43,065 at private nonprofit institutions, and $23,776 at private for-profit institutions.                                                                                                                      

4

Maldonado (2018) Price of College Increasing Almost 8 Times Faster Than Wages <https://www.forbes.com/sites/camilomaldonado/2018/07/24/price-of-college-increasing-almost-8-times-faster-than-wages/#1df0140366c1>`_                                                                                                                      

5

IRS Activity Code (B82 Scholarships & Student Financial Aid) and definition                                                                                                                      

6

Postsecondary is defined as any education beyond high school.                                                                                                                      

7

The National Center for Education Statistics defines college enrollment as the percentage of 18- to 24-year-olds enrolled as undergraduate or graduate students in two- or four-year institutions. College persistence is the percentage of students who return to college at any institution for their second year.                                                                                                                      

8

See Page and Scott-Clayton (2016) Improving college access in the United States: Barriers and policy responses for a summary of the literature.                                                                                                                      

9

Maggio et al (2019) Second Chance: Life without Student Debt find that student debt relief reduces indebtedness by 26 percent, reduces likelihood of defaulting on other accounts by 12 percent, increases borrowers’ geographical mobility and probability of changing jobs, and increases income by more than $4,000 over a three-year period (equivalent to about two months’ average salary). Forgiving $9,000 in debt increases income by about $4,000 over a three-year period.                                                                                                                                                                                                                                

10

Oreopoulos and Salvanes (2011) Priceless: The Nonpecuniary Benefits of Schooling                                                                                                                                                                                                

11

Angrist et al. (2016) Evaluating Post-Secondary Aid: Enrollment, Persistence, and Projected Completion Effects                                                                                                                                                                                                

12

Bartik et al. (2019) The Effects of the Kalamazoo Promise Scholarship on College Enrollment and Completion                                                                                                                                                                                                

13

Denning et al. (2019) ProPelled: The Effects of Grants on Graduation, Earnings, and Welfare                                                                                                                                                                                                

14

Anderson and Rab (2018) Experimental Evidence on the Impacts of Need-Based Financial Aid: Longitudinal Assessment of the Wisconsin Scholars Grant                                                                                                                                                                                                

15

Anderson (2019) When Financial Aid is Scarce The Challenge of Allocating College Aid Where it is Needed Most                                                                                                                                                                                                

16

Fack and Grenet (2015) Improving College Access and Success for Low-Income Students: Evidence from a Large Need-Based Grant Program                                                                                                                                                                                                

17

Sjoquist & Winters (2015) State Merit-based Financial Aid Programs and College Attainment                                                                                                                                                                                                

18

The Pell Grant program is the largest U.S. federal grant program for college students with demonstrated financial need.                                                                                                                                                                                                  

19

This study is the best (and only) available evidence that we are aware of to estimate the effect of scholarship aid on master’s degree completion rates. In France, the costs of postsecondary education are mainly driven by living expenses; tuition and fees at public universities are quite low. In contrast, in the U.S., tuition and fees are usually much higher for master’s degrees. Given the lack of available evidence from the U.S., we use this study despite its limited generalizability.                                                                                                                      

20

Four of the seven studies we rely upon utilize regression discontinuity designs — this means the treatment effects are specific to those students near the discontinuity. Using the treatment effects from these studies requires us to assume generalizability to a broader population of students than those near the specific cutoffs.                                                                                                                      

21

Merit based programs have an academic achievement requirement at the time of application — usually high school grade point average (G.P.A.) or being in a top percentile of the graduating high school class. However, many merit and need based programs also have college G.P.A. requirements for continued scholarship eligibility. So we use academic performance prior to award receipt to distinguish merit programs from other programs. Need based programs have an eligibility requirement related to financial need. Scholarship programs that require students to demonstrate both academic achievement and financial need at the time of application are referred to as “both.” The fourth category includes scholarships that are neither merit nor need based and instead target a broad group of students (e.g., all high school graduates from a school district). These are sometimes referred to as place-based scholarship programs although we use the term “neither (need nor merit based)” or “open eligibility.”                                                                                                                      

22

We assume that scholarships targeting niche groups (e.g., Irish American students, student athletes) are neither merit- nor need-based unless there is evidence to the contrary. We find that athletic scholarships are one common niche targeting. There are no studies estimating the effect of athletic scholarships on graduation rates for student athletes. However, according to the National Collegiate Athletic Association, student athletes tend to graduate at similar rates to nonathletes. Given the lack of additional evidence, we treat athletes like any other niche group.                                                                                                                      

23

There is some evidence that the difficulty of the scholarship application process can play a role in scholarship takeup and effects on graduation. We would ideally account for relative difficulty of the application process but are unable to do so due to limited public information on the scholarship programs under review. However, the difficulty of the application process is captured, albeit imperfectly, by our consideration of eligibility requirements: We assume merit and need based programs are inherently more difficult to apply for compared to scholarships that are neither merit nor need based.                                                                                                                      

24

We take a simple — not weighted — average of the relevant treatment effects when nonprofits provide a combination of scholarship types. In the future, we plan to explore applying best practices in meta-analysis to estimate weighted averages treatment effects.                                                                                                                      

25

For example, assume a nonprofit provides scholarships (neither merit nor need based) to 55 students and reports that 50 scholarship recipients graduated with a bachelor’s degree. Assume also that the nonprofit’s scholarships cover 82.5 percent of each student’s educational costs — the same as Bartik et al. Taking the graduation treatment effect from Bartik et al., we know that scholarships boosted graduation rates by 7.9 percentage points from a rate of 30 percent for non-recipients. Therefore 50*(0.3/0.379) = approximately 40 students who would have graduated in the counterfactual scenario or 10 students who graduated because of the nonprofit.                                                                                                                      

26

Treatment effect on bachelor’s degree graduation among students originally awarded scholarships to pursue associate’s degrees.                                                                                                                      

27

Treatment effect on enrollment in any type of college degree among students originally awarded scholarships to pursue associate’s degrees.                                                                                                                      

28

Scholarship coverage ratio for Denning et al. (2019).                                                                                                                      

29

Denning et al. (2019) only report the percent increase in community college enrollment due to eligibility for additional Pell aid. We combine this percent boost with the national average enrollment in community college for low income high school graduates to estimate the percentage point increase in enrollment. From there, we apply the national average associate’s degree completion rate to estimate the graduation effect. This gives us the effect of additional aid on the neediest Pell-eligible students, studied by Denning et al. However, Anderson and Rab (2018) find that for low income students as a whole, additional scholarship aid does not affect community college graduation rates. We therefore conclude that the effect of scholarships on low income community college students is driven by a graduation boost enjoyed by the lowest income needy students and not by relatively higher income needy students. So we take the average graduation treatment effects from Denning et al. and Anders and Rab to estimate the overall effect.                                                                                                                      

30

This review of 25 state merit aid programs finds no statistically significant effects on enrollment or graduation. So we assume zero treatment effect for both enrollment and graduation. Thus the impact of merit-only scholarship programs is only the income boost from the scholarship itself.                                                                                                                      

31

The treatment effect from Anderson and Rab (2018) is included in the adjusted treatment effect from Denning et al (2019) - see the footnote one before this. Sjoquist and Winters (2015) find no effects on graduation, so we assume zero effect of merit-only scholarship programs. Accordingly, merit-only scholarships are not included in our calculation of the scholarship coverage adjustment.                                                                                                                      

32

Bartik et al (2016) The Merits of Universal Scholarships: Benefit-Cost Evidence from the Kalamazoo Promise; Rothstein (2017) Measuring the Impacts of Teachers: Comment; Chetty et al. (2014) Where is the Land of Opportunity? The Geography of Intergenerational Mobility in the United States                                                                                                                      

33

Bartik and Hershbein (2018) Degrees of Poverty: The Relationship between Family Income Background and the Returns to Education <https://research.upjohn.org/cgi/viewcontent.cgi?article=1302&context=up_workingpapers>_ estimate that the return to a bachelor’s degree (compared to a high school diploma) is 70.6 percent for students from low income households. For students from high income households, the return to a bachelor’s degree (compared to a high school diploma) is 136 percent. We divide the two to estimate the proportional return for a low income student compared to a non-low income student.                                                                                                                      

34

Denning et al. (2019) use administrative data from the Texas Higher Education Coordinating Board and the Texas Workforce Commission to estimate the earnings returns for students eligible for additional Pell Grant aid. First time in college (F.T.I.C.) students receiving about $500 more in Pell aid are 3.3 percentage points more likely to graduate with a bachelor’s degree in 5–6 years. Assuming linearity, that translates to a 6.6 percentage point graduation boost for a $1,000 scholarship. The earnings boost from $1,000 in additional scholarship aid for F.T.I.C. students four to seven years after receiving it is, on average, $1,647. Using our methodology with C.P.S. data, the average annual earnings boost for a low-income (i.e., Pell-eligible) individual with a bachelor’s degree compared to a high school degree is $22,430. If we multiply this earnings differential by the 6.6 percentage point boost from Denning et al, the return is about $1,480. This is quite similar to the earnings return estimated by Denning et al. While we opt to use C.P.S. data in our earnings estimates, this comparison demonstrates that our approach is in alignment with the research literature.                                                                                                                      

35

Graduates may enjoy better jobs that come with benefits that they may not have enjoyed with the types of jobs they would get without the additional degree. In its cost-effectiveness methodology, the Robin Hood Foundation adds an estimate of fringe benefits (at 20 percent) to earnings estimates. We do not add such a premium due to lack of information on the types of jobs secured by graduates.                                                                                                                      

36

Levin et al. (2007) The Costs and Benefits of an Excellent Education for America’s Children                                                                                                                      

37

Bartik et al. (2016) The Merits of Universal Scholarships: Benefit-Cost Evidence from the Kalamazoo Promise                                                                                                                      

38

See Zimmerman (2014) The Returns to College Admission for Academically Marginal Students for a review of the research on education’s causal relationship to earnings.                                                                                                                      

39

Accounting for the increase in income enjoyed by scholarship recipients requires us to assume that the disbursement of a scholarship from Nonprofit A does not diminish the amount of scholarship assistance provided by any other scholarship program. Each scholarship nonprofit is an independent entity with its own programmatic resources, meaning Nonprofit A’s administration of scholarships should not have any effect on Nonprofit B’s provision of scholarships. The counterfactual assumption is that the amount of scholarship assistance provided by Nonprofit B would not be affected by the absence of Nonprofit A. This assumption implies that the benefit of a scholarship to a beneficiary in need constitutes a net gain; the gain is not offset by reductions in scholarship assistance provided to other beneficiaries.                                                                                                                      

40

We only count one academic year’s worth of the costs of college for students caused to graduate because of the counterfactual. Within the group of students caused to graduate because of the scholarship, there are two possible counterfactuals. First, in the absence of the scholarship some students would have simply stopped their education after completing the degree level down - an associate’s degree instead of a bachelor’s degree, for example. Second, some students would still pursue the same degree level but stop part way through pursuit of the bachelor’s degree, for example. The midpoint between these two counterfactuals is one additional year, so we count one year of costs.                                                                                                                      

41

Angrist et al. (2016)                                                                                                                                                                                                

42

Federal Student Aid (FSA) 2018 Annual Report                                                                                                                                                                                                

43

Postsecondary is defined as any education beyond high school.