This slideshow presents an examination of propensity score modelling and its potential for replicating the findings of a randomized controlled trial in evaluation research.
The presenters of this slideshow discuss propensity score modelling (PSM) and whether it may adequately replace randomized controlled trials (RCTs), which have been the ideal for methodological research but are not always possible in many criminal justice settings. This presentation reports on an investigation that tested the ability of PSM techniques to replicate the findings from 10 criminal justice RCTs. PSMs were designed to simulate the randomization of an RCT, and many criminal justice researchers, agencies, and funders, have demonstrated a preference for PSMs in evaluation research. The presenters note, however, that few cross-validation studies exist that test the ability of PSM to replicate the findings of an RCT, and previous research suggest mixed findings on the topic. The presenters’ investigation into whether PSM techniques were able to replicate RCT findings would determine whether PSM should continue to be used in criminal justice research. Before presenting the details of the analysis, the researchers provide their methodological overview: identify appropriate RCTs to test; introduce artificial selection bias, removing RCT balance; conduct several PSM analyses; compare balance of RCT and PSM techniques; examine outcomes, specifically the difference in PSM and RCT estimates; meta-analyze the results; and perform moderator analyses of important PSM traits. Results suggest that PSM can be an effective means for simulating an RCT experiment: PSM replicated the direction and magnitude of RCT findings to a high degree; magnitude of effective size (ES) differences from across PSM were small; and in five of the seven PSM comparisons, differences in d range were not statistically different than zero.