This report provides details of a research study which ultimately demonstrates that propensity score modeling can be a reliable and valid approach to estimating causal inference when conducting a randomized control trial experiment is impossible.
In this paper, the authors report on their efforts to resolve the critical question of whether or not propensity score modeling (PSM) methods can replicate the results from randomized control trials (RCTs) of criminal justice evaluations. The authors focused their efforts on seven different PSM techniques: one-to-one matching, with and without a caliper; one-to-many matching, with and without a caliper; inverse probability of the treatment weighting (IPTW); satisfied weighting scheme; and optimal pairs matching. For their research, the authors gathered datasets of 10 publicly available and restricted RCT studies from the National Archive of Criminal Justice Data (NACJD) which met two baseline criteria for randomization, they introduced an artificial selection bias into the treatment groups of those investigations, and then used each PSM technique to remove that selection bias. The authors describe their research methodology and comparison criteria of the RCT and PSM techniques and present their research results. They discuss how their investigation provides support for the use of PSM in criminal justice research and suggests that those seven PSM methods can be an effective way to estimate reliable and valid simulation of RCT experiments. Based on the findings discussed, the authors emphasize two critical points of caution for anyone using PSM in evaluation research: first, to test multiple techniques to gain confidence in the estimated effect sizes; and second, to assess balance using all appropriate methods and disseminate those assessments. The authors note their research limitations, and conclude that their cross-validation meta-analysis demonstrates clear evidence that PSM can be a reliable and valid approach to estimating causal inference in the absence of the ability to conduct an RCT experiment, however PSM does, in some instances, fall short of the RCT.