SENSITIVITY, BIAS, AND THEORY IN IMPACT EVALUATION (FROM EVALUATION STUDIES REVIEW ANNUAL, VOL 4, 1979, BY LEE SECHREST ET AL - SEE NCJ-65332)

• NCJ Number: 65334
• Author(s): R F BORUCH; H GOMEZ
• Date Published: 1979
• Length: 26 pages
• Contract Number(s): NIE-C-74-0115
• Sponsoring Agency: US Dept of Health, Education, and Welfare
• Corporate Author: Sage Publications, Inc
• Publication Format: Document
• Publication Type: Program/Project Evaluation
• Language: English
• Country: United States of America
• Note: REPRINTED FROM PROFESSIONAL PSYCHOLOGY, (NOVEMBER 1977), P 411-434

Annotation

TO DEVELOP MORE RIGOROUS AND SOCIALLY BENEFICIAL PROGRAM EVALUATION, THIS ARTICLE IDENTIFIES AND ATTEMPTS TO SOLVE MEASUREMENT PROBLEMS WHICH, IN USUAL EXPERIMENTAL DESIGN, INVITE UNDERPOWERED (INSENSITIVE) EXPERIMENTS.

Abstract

MEASUREMENT SHOULD BE CONCERNED NOT ONLY WITH THE RELIABILITY OF THE DEPENDENT VARIABLE BUT, MORE IMPORTANTLY, WITH VALIDITY OF MEASUREMENT AND WITH MEASUREMENT, THROUGH SYSTEMATIC OBSERVATION, OF THE TREATMENT VARIABLE. IN AN IMPACT EVALUATION, WHICH ENTAILS THE MEASUREMENT OF THE RELATIVE EFFECT OF A PROGRAM ON ITS TARGET GROUP, EVALUATORS SHOULD BE ABLE TO ANTICIPATE THE POWER OF AN EVALUATION DESIGN, ENHANCE THAT POWER, AND UNDERSTAND THE EVALUATION RESULTS. WHEN A PROGRAM IS TRANSFERRED FROM THE LABORATORY TO THE FIELD, THE PRECIPITOUS DROP IN STATISTICAL POWER RESULTS FROM THE MULTIPLICATIVE EFFECTS OF INVALIDITY OF RESPONSE AND INFIDELITY OF TREATMENT. TO DEAL WITH THIS PROBLEM, ORTHODOX MEASUREMENT THEORY SHOULD BE AUGMENTED BY INPUT FROM PROGRAM MANAGERS, DEVELOPERS, AND THE SUBSTANTIVE THEORIST. THE RESPONSE VARIABLE MUST BE MEASURED USING CRITERIA RELEVANT TO THE TREATMENT VARIABLE. CRITERIA FOR DETERMINING RELEVANCE INCLUDE SUBSTANTIVE THEORY, PRIOR DATA, PILOT TESTS, SIDE THEORIES, AND OTHER MODELS FOR UNDERSTANDING THE RESPONSE VARIABLES' RELATIONSHIP TO THE ACTUAL STATUS OF THE INDIVIDUAL AND TO PROGRAM CONTENT. THE TREATMENT VARIABLE MUST ALSO BE EXAMINED, RATHER THAN ASSUMING THAT THE PRESCRIBED LEVEL OF TREATMENT IS EITHER PRESENT OR ABSENT. THE THREE MAIN PROBLEMS IN IDENTIFYING AND MEASURING TREATMENT IN FIELD STUDIES ARE (1) POLICY PACKAGED AS PROGRAM, (2) THE STRUCTURALLY INCOMPLETE PROGRAM, AND (3) INCOMPLETE PROGRAM RECEPTION OR DELIVERY AT THE INDIVIDUAL LEVEL. PILOT TESTS OR SIDE STUDIES, MOUNTED PRIOR TO A FULL-SCALE EVALUATION, SEEM WELL JUSTIFIED HERE. IT IS CONCLUDED THAT ORTHODOX EXPERIMENTAL DESIGN AND QUASI-EXPERIMENTAL APPROACHES NEED TO BE AUGMENTED FOR EVALUATING SOCIAL PROGRAMS AND THAT AN INTERDISCIPLINARY TEAM APPROACH SHOULD BE USED IN EVALUATION. REFERENCE NOTES, A REFERENCE LIST, AND AN APPENDIX PRESENTING THE ALGEBRAIC FORMULATION OF THE MEASUREMENT THEORY, ARE INCLUDED. (CFW)