This study, under the Comprehensive Regional Information-Sharing Project (CRISP), examined the use of metrics as a tool to assess the effectiveness of a law enforcement information-sharing system (ISS) and its impact on operations.
Metrics are a set of measures that may be used to assess the success or failure of a system or program. Difficulties arise when trying to use metrics to assess system or program features that produce qualitative results which presents a challenge of measuring results that may not be quantifiable. The primary effort of this study (the Comprehensive Regional Information-Sharing Project (CRISP), performed by Noblis’ Center for Criminal Justice Technology in partnership with the U.S. Department of Justice, National Institute of Justice, was to devise a detailed, automated approach for developing a metrics collection and analysis program. The plan described, based upon a mapping between information-sharing system (ISS) objectives and potential metrics, includes the following steps: (1) define ISS program objectives; (2) determine which types of metrics to collect; (3) determine feasibility of the metrics; (4) map ISS program objectives and metrics; (5) collect metrics; and (6) analyze metrics collected. Recommendations for a metrics collection program resulting from the study include: (1) institute a formal plan for metrics collection; (2) consider the benefits of a preliminary behavioral study on how best to obtain quality input from users; (3) recognize the significant value of qualitative information as metric data; (4) use a combination of metrics to assess each objective; (5) leverage the relationship between law enforcement agencies and the broader criminal justice system; (6) acknowledge that some metrics will provide an indication of the usefulness of the ISS rather than identifying definitive relationships between ISS use and meeting of ISS objectives; (7) recognize that planning for and implementing a metrics collection program is a long-term process; (8) use ISS metrics as a design tool to plan for, evaluate, or improve other law enforcement programs; and (9) consider expanding this research to examine ISS and non-ISS programs beyond the criminal justice system. Tables and appendixes A-B