On Developing Evaluation Designs - A Summary Report

• NCJ Number: 92370
• Date Published: 1983
• Author(s): J M Tien
• Length: 21 pages
• Grant Number(s): 81-IJ-CX-K026
• Agencies: NIJ-Sponsored
• Publication Type: Program/Project Evaluation

Annotation

This guide to program evaluation design identifies a framework which links program characteristics to design elements, defines five related design components which contain the essential design elements, and develops a linear statistical model which highlights some of the key underlying issues in program evaluation.

Abstract

The design framework, which assumes that the program and its evaluation design are developed concurrently, is based on a dynamic roll-back approach consisting of three steps leading to a valid and comprehensive evaluation design. The sequence rolls back in time from (1) a projected look at the range of program characteristics to (2) a prospective consideration of the threats to the validity of the final evaluation and (3) a more immediate identification of the evaluation design elements. The five related design components are test hypotheses, selection scheme, measures framework, measurement methods, and analytic techniques. The test-hypotheses component includes the range of issues leading up to the establishment of test hypotheses, while the selection-scheme component involves the development of a scheme for the selection and identification of test groups and, if applicable, control groups, using appropriate sampling and randomization techniques. The measures-framework component involves specifying the set of evaluation measures that is to be the focus of the evaluation and the construction of a model that reflects the linkages among the measures. The list of issues and elements constituting the measurement-methods component includes measurement time frame, measurement scales, measurement instruments, measurement procedures, measurement samples, measurement quality, and measurement steps. Analytic techniques are used to conduct statistical tests of significance; combine, relate, or derive measures; assist in the evaluation; provide data adjustments for nonequivalent test and control groups; and to model test and control situations. Twelve bibliographic entries are provided.