ON RANDOMIZED EXPERIMENTS, APPROXIMATION TO EXPERIMENTS, AND DATA ANALYSIS (FROM EVALUATION RESEARCH METHODS - A BASIC GUIDE, 1977, BY LEONARD RUTMAN - SEE NCJ-49434)

• NCJ Number: 49440
• Author(s): R F BORUCH; D RINDSKOPF
• Date Published: 1977
• Length: 35 pages
• Grant Number(s): NIE-C-74-0115
• Sponsoring Agency: US Dept of Health, Education, and Welfare
• Corporate Author: Sage Publications, Inc
• Publication Format: Document
• Publication Type: Report (Study/Research)
• Language: English
• Country: United States of America

Annotation

DATA-ANALYTIC JUSTIFICATIONS FOR RANDOMIZED EXPERIMENTS ARE IDENTIFIED ALONG WITH PROBLEMS ARISING FROM NONEXPERIMENTAL DESIGNS. APPROACHES FOR METHOD APPRAISAL AND CATEGORIES FOR DATA HANDLING ARE DISCUSSED.

Abstract

PRELIMINARY DEFINITIONS OF TERMS AND CONCEPTS ARE OUTLINED RELATIVE TO THE DISCUSSION OF PROBLEMS IN ESTIMATING THE EFFECTS OF A NEW SOCIAL PROGRAM ON PARTICIPANTS. THE FOCUS IS ON OBTAINING UNBIASED ESTIMATES OF A PROGRAM'S EFFECTS BY ESTABLISHING A REASONABLE BASIS FOR DETERMINING THE WAY PROGRAM PARTICIPANTS MIGHT HAVE BEHAVED HAD THEY NOT PARTICIPATED IN THE PROGRAM. RANDOMIZED EXPERIMENTAL DESIGNS PROVIDE SUCH A BASIS FOR THIS KIND OF STUDY, AND THREE KINDS OF TECHNIQUES FOR THE DESIGNS ARE EXAMINED; (1) PROCEDURAL STRATEGIES, (2) EMPIRICAL STRATEGIES, AND (3) RHETORICAL APPROACHES SPECIFYING THE NULL CONDITION BY ASSUMPTION. SIMPLE COMPARATIVE TACTICS FOR APPRAISING METHODS OF ANALYSIS ARE: COMPARISON OF ESTIMATES FROM RANDOMIZED EXPERIMENTS AND APPROXIMATIONS OF EXPERIMENTS; DRY-RUN EXPERIMENTS AND MULTIPLE PRETESTING; AND THE RATING OF INNOVATIONS AND MAJOR THREATS TO VALIDITY. REGARDLESS OF THE STRATEGY USED IN CONJUNCTION WITH RANDOMIZED EXPERIMENTS, DATA STEMMING FROM STUDIES MUST BE ANALYZED. SEVERAL TECHNIQUES ARE SUGGESTED, AND THOUGH NONE GUARANTEES UNBIASED ESTIMATES OF PROGRAM EFFECTS, ALL HELP TO ESTABLISH THE PLAUSIBILITY OF ANY GIVEN INTERPRETATION OF THE DATA. THE CHECKLIST APPROACH ENTAILS THE MAKING OF JUDGMENTS ABOUT THE EXISTENCE OF AN EFFECT, AND ENUMERATING ALL THE COMPETING CAUSES OR EXPLANATIONS FOR THE EFFECT OTHER THAN THE PROGRAM ITSELF. AN APPROACH BASED ON STEROTYPICAL ERRORS IDENTIFIES THE WAYS IN WHICH INFERENCES ABOUT A PROGRAM IMPACT MAY BE WRONG AND THEN NARROWS THE LIST DOWN. REGRESSION EFFECTS, DIFFERENTIAL GROWTH, AND FLOOR EFFECTS ARE THE MAJOR FACTORS DISCUSSED. MULTIPLE ANALYTIC MODELS AND METHODS OF ANALYSIS ARE EXAMINED. THESE ALLOW NUMERICAL ESTIMATION OF THE IMPACT OF A PROGRAM. FIVE ANALYTIC CATEGORIES ARE CONSIDERED: (1) BLOCKING OR MATCHING; (2) STANDARDIZATION; (3) CONVARIANCE METHODS; (4) REGRESSION APPROACHES AND (5) STRUCTURAL EQUATIONS. REFERENCES ARE PROVIDED. (DAG)