This resource presents the National Institute of Justice (NIJ) Recidivism Forecasting Challenge, and provides an overview of the metrics used to judge the entries and contextualizes how the winners’ forecasts performed in terms of accuracy and fairness.
The National Institute of Justice (NIJ) recently hosted the Fiscal Year 2021 Recidivism Forecasting Challenge. The primary aim of this research competition was to increase public safety and the fair administration of justice by improving the ability to forecast and understand the variables that impact the likelihood that an individual under parole supervision will recidivate. Entrants were provided with two datasets. The first was a training dataset of over 18,000 individuals released from prison to parole supervision in the state of Georgia during the period of January 1, 2013, through December 31, 2015. These data contained information about individuals’ demographic characteristics, supervision case information, prison case information, prior criminal and community supervision history in the state of Georgia, activities for current supervision, and whether they recidivated in any of the 3 years after they began supervision. The second was a test dataset (n = 7,807) used to develop models for forecasting the probability that an individual on parole will recidivate within their first, second, or third year on parole. For each of the Challenge’s three submission periods, models were scored by two indices: (1) a Brier score, which is a measure of accuracy, and (2) fairness and accuracy via a difference in the false positive rate between Black and white racial groups in conjunction with the Brier score. Prizes were awarded to the entries that had the lowest error in the forecasts for males and females, and the average of these two scores. Additionally, prizes were awarded to the entries that had the highest fairness and accuracy scores after any assessed fairness penalties. In order to put these results into context, this paper compares the winning results to a variety of naive models, such as predicting recidivism by random chance or using the average recidivism rate by population demographic for those in the sample. Naive demographic models outperformed the chance model, and submitted forecasts outperformed the best naive demographic models. This suggests that more advanced algorithms have improved the capability of determining which variables accurately forecast recidivism. Improved algorithms could assist community corrections agencies in identifying and prioritizing the needs of those on parole and promoting more successful reintegration into society.