The recent changes that the UK government imposed on the Peterborough social impact bond program are interesting. On the one hand, they illustrate the risk faced by policy innovators seeking to pilot new programs: A change in government policy may reduce their ability to test key hypotheses of their model. On the other hand, they show that policy innovations can fall victim to their own success.
The change
The UK Ministry of Justice just introduced Transforming Rehabilitation, a program that makes wide-ranging changes to the government’s effort to reduce recidivism. Some changes scale up components of the Peterborough SIB to all eligible released prison inmates in the country. This scale-up complicates an independent evaluator’s effort to determine the impact of the program by using a comparison group composed of released inmates that are similar to the ones in the Peterborough SIB except for their receiving the program. Since all eligible released inmates will receive some Peterborough-like services, a rigorous counterfactual cannot be created. (Although less precise counterfactuals are still available.)
Social Finance, which created this SIB, describes the changes: “The proposal will enable the Peterborough intervention (the “One Service”) to continue but will change the way the service is funded and remove the outcomes payments for the third and last cohort of prisoners to be released from June 2014. Details of the alternative funding arrangement are still being discussed but the Ministry of Justice is keen to ensure that the same level of rehabilitation support continues to be provided to this group through until the new regime is established and a new provider is in a position to establish supply chain arrangements for rehabilitation.”
Victim to its own success
The UK government’s new approach to reducing recidivism contains components that resemble those introduced in the Peterborough social impact bond. The most similar ones are:
- “A nationwide ‘through the prison gate’ resettlement service will be put in place, meaning most offenders are given continuous support by one provider from custody into the community. We will support this by ensuring that most offenders are held in a prison designated to their area for at least three months before release.” (Source)
- “New payment incentives for market providers to focus relentlessly on reforming offenders will be introduced, giving providers flexibility to do what works and freedom from bureaucracy, but only paying them in full for real reductions in reoffending.” (Source)
The first component resembles the case management approach that is used in the Peterborough and in other recidivism SIBs, where rehabilitation services staff engage inmates just before they leave prison or meet them at the prison gate and enroll them into several services customized to fit their needs. (The exact timing of when inmates are engaged depends on the agreement worked out with the correctional facility.)
The second component comes straight out of the pay for success playbook: Providers get flexibility to alter their program and the incentive to make alterations that improve the program’s intended impact of reducing recidivism.
Although I have not read enough about Transforming Rehabilitation to say that the Peterborough pilot made a strong contribution to the innovative components of that initiative, Toby Eccles of Social Finance thinks that the pilot played a hand.
Harder to test key hypotheses
Like many new approaches to solving social and policy problems, SIB designers are being judged on some fairly obvious questions: Does the new approach work better than the old approach? Do the people implementing the new approach prefer it? Some of these questions will now be harder to answer because the pilot will not be allowed to run its full course: We will lose all the learning we would have received from the last two years of the program. The original investor agreement will have to be revised. And the government’s use of the pilot becomes less clear.
I divide all this into three questions below.
1. Will the program produce outcomes that are better than the ones we are currently seeing?
So far this question has not been definitively answered, and the government’s change of the SIB makes the question more difficult to answer.
According to the data that has been released by the UK government, the Peterborough pilot has delivered good results for the first of its three cohorts. Over the past 12 months, the government has measured the number of reconviction events, which happen when a person commits an offense and is convinced for that offense in court either during those 12 months, or in the following 6 months. The government tracked reconviction events of the 1034 inmates released from Peterborough prison. Over the 12-month period, the number of reconviction events per 100 inmates fell from 159 to 141, or 11%. For comparison the government tracked the corresponding national figure, which saw an increase of 10% over the same period of time. This means that the Peterborough pilot reduced one measure of recidivism by 21% so far.
However, the government will use a different comparison group in its final evaluation of the program. In that evaluation, the government will use a control group of released inmates that are matched to the group in the Peterborough program according to a set of characteristics. This control group creates a better counterfactual scenario for Peterborough than does the overall national inmate population because it includes only released inmates that are very similar to ones that are part of the Peterborough program.
With the new policy, the government made it impossible to evaluate the SIB in the third cohort, and this is the main reason that the third cohort is eliminated. The government will still be able to make a statement about the SIB’s ability to improve social outcomes, but the statement is now less powerful because a third of the program’s learning is lost.
The good news is that Peterborough is not the only SIB that can be used to answer this important question. Many than 40 SIBs have followed Peterborough, and many include thoughtful evaluation designs that will be able to provide evidence for this question.
2. What value will the government find in the program — reduces savings to its budget, improved ability to fund risky interventions, improved outcomes, or something else?
It is difficult to tell what value the government found in the Peterborough SIB so far.
If the government sought to test changes to its recidivism policies and implement ones that worked, then the test seems incomplete. Although process evaluations by RAND and others suggest that people like the changes, the outcome evaluation has not been completed, published or discussed. For the government to integrate Peterborough changes into its Transforming Rehabilitation program based on the success of those changes, it must make the argument that its preliminary assessment of outcomes is showing that the impact is so large that it cannot wait for the final outcome evaluation but rather must scale up the changes now. The Peterborough SIB is showing promise, but I have not seen anyone argue that its results are so good that we can build policy around them.
One of the original arguments for SIBs said that government can use them not only to move toward preventative services, but also to save money by preventing expensive conditions such as incarceration, homelessness or asthma. For example, it may cost $40,000 to keep a person in prison for a year, but only $20,000 to prevent that person from entering prison. However, initial pilots cannot make the cost-benefit argument as easily because they are very expensive to set up, especially when they include costly evaluations designed to test the pilots’ efficacy. In addition, we have seen government express interest in SIBs even when they do not create savings that are easy to calculate or easy to convert into explicit budget savings.
Perhaps the most plausible scenario is that the government’s frustration with high recidivism rates gave rise both to the Peterborough pilot and to the burning desire to transform the entire rehabilitation program. The government did not want to wait for Peterborough to conclude and, while the pilot informed its transformation initiative, but it was also swamped by the government’s broader efforts.
This suggests that future SIB pilots face a double edged sword: Policy innovators should focus their pilots on pressing social issues, but pressing social issues are also ones that are likely to see more policy change that would threaten the structure of the pilot program. So perhaps the value of a SIB pilot is to inform a dynamic policy conversation, and therefore these pilots should be designed to provide answers faster.
3. What value will investors find in the program?
Investors are probably harmed more by this policy change than are any other participants in the SIB. (The beneficiaries of SIB services, for example, will continue to receive services, and they may even continue to receive services from the same group of providers as they do now.) Therefore, this policy change may chill investor interest in the SIB just as other efforts are bringing more and more types of investors to the table.
About 17 individuals and institutions invested approximately $8 million for 6 years of service delivery in the Peterborough SIB. They expect to receive principal and interest payments that are based on the program’s performance in years 2, 4 and 6. Now the last cohort, in years 5 and 6, will not be subject to a performance based payment. How does this affect investors’ interest payments? If any of the original interest payments are tied to the programs’ results in years 5 and 6, they must be revised.
One option is to tie payments to the programs’ results in years 1-4. But if service providers would have improved their results in years 5 and 6, then the new interest payments tied to years 1-4 will be lower because they do not benefit from those improvements. And improvement of results over time is a reasonable expectation, especially if providers are incentivized financially. For example, providers may become more efficient in delivering outcomes over time, or shift their spending toward activities are produce better results.
Another option is to tie interest payments to the program’s results in years 5 and 6 and use a different way to evaluate results in those years (since the original evaluation of creating a matched control group is unavailable). But it seems that the government has ruled out this option: “However, maintaining the PbR element of the scheme at Peterborough until 2017 for the third and final cohort is not possible, as the majority of prisoners within that group will already be receiving 12 months supervision and rehabilitation as a result of the wider reforms to probation.”
Investors will soon know answers to these questions — and I hope that some of the answers eventually become public so that the policy innovators around the world can learn how policy changes like this one will affect their own payment by results programs.
Michael Belinsky 