Beyond Random Assignment - Factors Affecting Evaluation Integrity

• NCJ Number: 75886
• Journal: Evaluation Review Volume: 5 Issue: 1 Dated: (February 1981) Pages: 51-67
• Author(s): E L Rezmovic; T J Cook; L D Dobson
• Date Published: 1981
• Length: 17 pages
• Publication Format: Article
• Language: English
• Country: United States of America
• Note: Portions of this article were presented as a paper at the Midwestern Political Science Association Convention, Chicago, Illinois, April, 1979 and the Midwestern Psychological Association Convention, Chicago, Illinois, May 1979.

Annotation

Experience gained from an experimental evaluation of a Chicago program serving ex-offenders is reported and used as the basis for a discussion of practical problems in maintaining the integrity of evaluation studies.

Abstract

These problems revolved around budgeting, randomization, the data collectors, interview payments, and evaluation of black box treatments. The program dealt with ex-offenders' employment and social/personal needs. The program evaluation focused on the program's social reintegration component, in which about 1,000 parolees per year were each assigned to 1 of 3 treatment models. The models included a staff associate model, in which private citizens were matched with individual ex-offenders for a 1-year period; a VISTA model, in which VISTA volunteers who were ex-offenders were assigned caseloads of 15 to 20 clients for 3-month intervals; and a paraprofessional model, in which employed paraprofessionals handled caseloads of 25 to 30 clients for 6-month periods. Clients were randomly assigned to one of the treatment models in order to facilitate the evaluation. The inclusion of evaluation funds in the program's overall operating budget was designed to strengthen both the program and the evaluation but created a sense of competition for scarce resources. The random assignment process was not always followed, due to staff errors and lack of understanding of the assignment process. Use of data collectors not closely allied to the research effort resulted in some fraudulent reporting of interviews because of data collectors' frustration with the task level, personal problems, and concrete benefits of cheating. Compensation of research subjects had both benefits and problems. Through monitoring of treatment implementation was impossible, resulting in a 'black box' effect. Overall, a strong experimental design cannot stand alone, and nontechnical aspects of evaluation can significantly affect the validity of research results. Both research experience and research methodology should be communicated when evaluation results are reported. Fifteen references are listed.