USING THE RESULTS OF RANDOMIZED EXPERIMENTS TO CONSTRUCT SOCIAL PROGRAMS - THREE CAVEATS (FROM EVALUATION STUDIES REVIEW ANNUAL, VOL 4, 1979, BY LEE SECHREST ET AL SEE NCJ-65332)

• NCJ Number: 65342
• Author(s): D B PILLEMER; R J LIGHT
• Date Published: 1979
• Length: 10 pages
• Corporate Author: Sage Publications, Inc
• Publication Format: Document
• Publication Type: Program/Project Evaluation
• Language: English
• Country: United States of America

Annotation

THREE CAVEATS RELATING TO THE USE OF RESULTS OF RANDOMIZED EXPERIMENTS TO CONSTRUCT SOCIAL PROGRAMS ARE PRESENTED.

Abstract

ALTHOUGH RANDOMIZED, CONTROLLED FIELD TRIALS ARE USEFUL IN CONDUCTING SOCIAL POLICY RESEARCH, RANDOMIZATION DOES NOT BY ITSELF ENSURE GOOD SOCIAL POLICY INFERENCES. IN SEVERAL SITUATIONS, THREE OTHER FACTORS MAY RESULT IN SITUATIONS IN WHICH A WELL-EVALUATED EXPERIMENTAL TREATMENT MAY PERFORM UNPREDICTABLY IF IMPLEMENTED MORE WIDELY. THE FIRST FACTOR IS THE CONCEPT OF RELATIVE GAIN, IN WHICH THE BENEFIT A PROGRAM CONFERS TO ANY ONE RECIPIENT IS A FUNCTION OF HOW MANY OTHER PEOPLE RECEIVED THE PROGRAM. IN SUCH SITUATIONS, WIDE EXPANSION OF A PROGRAM WILL NOT PRODUCE GAINS TO RECIPIENTS. SECONDARY ANALYSIS CAN GIVE A PRELIMINARY INDICATION OF WHETHER A PARTICULAR PROGRAM HAS A RELATIVE GAIN CONSTRAINT. IN SOME SITUATIONS, HOWEVER, ABSOLUTE GAINS ARE MORE IMPORTANT THAN RELATIVE GAINS. A SECOND POTENTIAL HAZARD IS JUDGING A PROGRAM'S SUCCESS BEFORE OBSERVING LATER PERFORMANCE. INITIAL GAINS MAY LATER DISSIPATE, OR INITIAL FAILURE MAY BE FOLLOWED BY LATER SUCCESS. CONTROLLED, RANDOMIZED FIELD TRIALS CANNOT ELIMINATE THE DANGERS OF PREMATURELY JUDGING A PROGRAM. EVALUATING SUBJECTS' PERFORMANCE LONGITUDINALLY IS THEREFORE OF CRITICAL IMPORTANCE. A THIRD PROBLEM IS THAT PROGRAMS' EFFECTS MAY DIFFER ACCORDING TO THE LENGTH OF TREATMENT. QUALITATIVE DIFFERENCES IN TREATMENT EFFECTS MAY DEPEND ON WHETHER TREATMENT IS TEMPORARILY SUPERIMPOSED ON NATURALLY EXISTING BEHAVIORS OR WHETHER IT FORCES THE REORGANIZATION OF THOSE BEHAVIORS. SHORT-TERM EXPERIMENTS, HOWEVER WELL DONE, DO NOT PROVIDE CONCLUSIVE EVIDENCE ABOUT LONG-TERM IMPLEMENTATIONS OF SIMILAR TREATMENT. WHEN LONG-TERM IMPACT IS UNCERTAIN, PROGRAMS SHOULD BE FREQUENTLY MONITORED FOR UNINTENDED LONG-TERM CONSEQUENCES. THESE THREE METHODOLOGICAL CAVEATS ALL SUGGEST A CENTRAL ROLE FOR SECONDARY ANALYSES OF EXPERIMENTAL STUDIES IN FORMULATING SOCIAL POLICIES. COMPARING MAJOR STUDIES IN AN AREA NOT ONLY HELPS IDENTIFY SITUATIONS WHERE EXPERIMENTAL STUDIES' POLICY IMPLICATIONS ARE UNCLEAR BUT ALSO CAN IMPROVE PREDICTIONS OF SUCCESS OR FAILURE IN THESE SITUATIONS. CASE EXAMPLES ILLUSTRATING THESE POINTS, NOTES, AND A REFERENCE LIST ARE INCLUDED. (CFW)