When Does Pay-for-Success Make Sense?

There is growing excitement about the potential of pay-for-success (PFS) financing, which allows independent investors to finance social interventions aimed at issues or problems of interest to the public sector. If a PFS project achieves pre-determined outcomes, the public sector will repay the private investors, potentially with a return on the investment.

The number of pay-for-success projects in the United States and worldwide continues to rise, with applications in an expanding array of human service, public health, education, employment, community corrections, and environmental interventions. Some examples of launched PFS interventions include supportive housing, post-incarceration job training, early childhood education, and nurse home-visiting for pregnant women. Our research shows significant potential for PFS to address the social determinants of health and wellness, which the World Health Organization defines as “the conditions in which people are born, grow, live, work and age.”

In this nascent field, however, there is little consensus regarding what makes an intervention appropriate for PFS. Some believe an intervention should have a strong evidence base for effectiveness, including randomized trials and cost analyses, before being considered for PFS financing. They argue that if the intervention’s effects and costs relative to benefits are unknown, a performance-based contract and payout schedule cannot be crafted without a huge amount of guessing and wishful thinking. Others argue that these academic standards of evidence are too high and unrealistic for PFS application in the real world. They argue that “evidence-informed” or theoretically plausible interventions with some promising pilot information should be sufficient for PFS financing. A third perspective holds that PFS financing is appropriate for innovative and unproven interventions as long as an investor is willing to take the risk.

Research Evidence Is Crucial

We side with the first of these groups, and here’s why. A long list of social programs and interventions have been implemented with great enthusiasm yet have not achieved their goals. The Drug Abuse Resistance Education (DARE) program is a prime example. This school-based education and prevention program spread throughout the 1980’s to the majority of public elementary schools in the United States with great conviction of effectiveness. However, subsequent research demonstrated no impact on the program’s desired outcomes. In fact, some evaluations concluded that children exposed to DARE were more likely to experiment with cigarettes and alcohol.

It’s no secret that interventions can have unintended consequences and that some actually cause harm. Program evaluators know that you cannot be sure a program is working as intended without conducting high-quality research. Without strong evaluations, you cannot be certain about an intervention’s strength or degree of impact—information essential to structuring a PFS deal.

It is difficult to justify the resources, risks, and opportunity costs of PFS initiatives when an intervention has no evidence base or when existing evidence raises red flags about the impact of a program. In addition, it’s concerning when unproven programs or services are given to low-income or otherwise socially vulnerable populations—groups the majority of PFS interventions target.

Based on our work to date, we propose a set of seven criteria for interventions being considered for PFS financing:

1. The intervention must address a problem of interest to the public sector.

2. The intervention must have a strong research evidence base in terms of effectiveness in a clearly identified population(s).

Taken together, these criteria demand compatibility between a public-sector problem of interest and an evidence-based program. We believe randomized controlled trials (RCTs) provide the strongest evidence for the direction and strength of an intervention’s effectiveness. However, RCTs are challenging, expensive, and often lead to concerns about generalizability. The more tightly controlled the study design, the more you have to worry about generalizing the results beyond the study population. Quasi-experimental research approaches (evaluation approaches in which there is a comparison group without randomization) can also provide strong evidence of intervention effectiveness. This is accomplished by strengthening research design elements such as sample size, adequacy of the control group, and relevancy of outcome measures used to evaluate the program.

3. The intervention must be economically attractive to the public sector.

This includes research evidence of cost savings (the intervention costs less than the public savings generated) or cost effectiveness (the outcomes achieved are of great enough value and worth the economic investment by the public sector). Understanding the economic implications of interventions is important to crafting a PFS contract, and requires that the evaluation research demonstrate both the impact of the intervention and its cost implications for the public sector. For example, high-quality research has shown that improving both the home physical environment and medical management of children with asthma can reduce expensive trips to the emergency room and hospitalizations. Understanding the cost of the intervention and how it reduces public health care expenditures are essential pieces of information in the design of a PFS initiative.

4. Outcomes must be expressed as metrics that are clearly defined and quantifiable.

Since PFS contracts are performance-based, outcome metrics that define success must be well-defined and measurable. Furthermore, the data necessary for measuring and analyzing outcomes in an evaluation must be high-quality and readily available. For example, the Hillside Intensive Community Asset Program (ICAP) in New York has a proven model and was considered for PFS financing. However, the project was unable to launch due to the inability to collect consistent data on program outcomes across various government departments.

5. Outcomes must be achievable in a reasonable and clearly understood time period.

The nature of prevention efforts is that the payoff is in the future. Prevention efforts for which the benefit to the public sector is many years away might not be attractive to public officials or investors.

For example, there is strong evidence that interventions that prevent smoking in children and adolescents will reduce health care costs in later life. This is not ideal for PFS financing, however, because investors do not want to wait decades to receive a payout on their investment. Instead, a PFS project could focus on proximate behavioral outcomes (such as smoking rates among adolescents or young adults) as more-immediate performance metrics.

6. The evidence-based interventions should be able to be implemented without significant administrative challenges.

Interventions must be implemented with fidelity to their designs and consistency to achieve intended outcomes, not using shortcuts or people without appropriate training. For example, multisystemic therapy (MST), an evidence-based intervention that reduces recidivism and out-of-home placement for youth involved with the juvenile justice system, focuses on youths’ families and connects to the educational, social welfare, and other systems that surround them. Given the intensity of the intervention, caseloads for MST workers are relatively low. Social service agencies must have the capacity to hire, train, and supervise the requisite number of MST professionals. MST also requires a strong working relationship with the juvenile justice system. This means that although MST is a strong and evidence-based intervention with great potential for PFS, it should only be implemented in settings that allow fidelity to its high-quality design.

7. An intervention’s implementation should face no significant political or stakeholder challenges.

An intervention could meet the other criteria described above but not appeal to stakeholder groups like government officials for political, moral, or other reasons. In other words, it is possible to identify an intervention that has a strong evidence base, would provide economic value to the public sector, and be implemented without significant administrative or logistical challenge, but one or more stakeholder groups may still oppose it. For example, an evidence-based teen pregnancy intervention that would increase women’s access to long-acting reversible contraceptives may meet with moral or religious objections from elected officials, public agency leadership, or community-based groups. As another example, some people object to supportive housing interventions that combine subsidized/free housing with social services to people with active substance abuse problems, because they believe taxpayers should not be providing housing support to people who use illegal drugs.

Refocusing Around Intervention Quality and Impact

Several PFS initiatives—including the Adolescent Behavioral Learning Experience Project for Incarcerated Youth in New York City—have failed to yield an investor payout, leading some to question “Does PFS work?” We find this question off point; PFS is a financing model, not an intervention. The “does it work” question should be focused on the quality and impact of the interventions selected for a PFS performance-based contract, not the model itself.

Intuition and wishful thinking are not substitutes for high-quality intervention research.

As communities and organizations working with PFS know, these projects are complex, and involve a great deal of planning and resources. Our proposed set of criteria is complimentary to six components of the Project Assessment Tool developed by the Urban Institute’s Pay for Success Initiative. That tool was designed to promote “good practices” in defining social problems, selecting strong interventions, designing and implementing PFS projects, and conducting rigorous evaluations. Our position is also in line with the Pew-MacArthur Results First Initiative, a project through which technical assistance and support is offered to states interested in implementing an evidence-based, cost-benefit analytical approach to investing in programs and policies. The main premise of the initiative is that rigorous use of evaluation research findings (including cost analyses) will increase understanding of which interventions are likely to work in specific populations. In turn, this will assist states in prioritizing programs and allocating resources.

We see great potential in the PFS financing model to spread effective interventions that address the social determinants of health—but only if the interventions have a solid evidence base to support the outcomes and time frames for success payments. A primary purpose of PFS financing is to provide value to the public sector. This will not happen if interventions do not meet their goals or positively impact participants. Intuition and wishful thinking are not substitutes for high-quality intervention research. The PFS field is becoming more sophisticated in its evaluations of launched PFS projects. It is time for PFS project partners to become more sophisticated in using evaluation and research literature to select appropriate, effective, and winning interventions before they start.

Read more stories by Paula Lantz & Samantha Iovan.