EVALUATION OF BROAD-AIM PROGRAMS - A CAUTIONARY CASE AND A MORAL (FROM READINGS IN EVALUATON RESEARCH, 2ND ED., 1977, BY FRANCIS G CARO - SEE NCJ-50468)

• NCJ Number: 50478
• Author(s): R S WEISS; M REIN
• Date Published: 1977
• Length: 10 pages
• Corporate Author: Russell Sage Foundation
• Publication Format: Document
• Publication Type: Program/Project Evaluation
• Language: English
• Country: United States of America
• Note: REVISION OF PAPER PRESENTED AT THE AMERICAN ACADEMY OF ARTS AND SCIENCES CONFERENCE ON EVALUATION OF SOCIAL ACTION PROGRAMS, MAY 2-3, 1969 REPRINTED FROM THE ANNALS OF THE AMERICAN ACADEMY OF POLITICAL AND SOCIAL SCIENCE, V 385 (SEPTEMBER 1969), P 133-142

Annotation

THE ADMINISTRATIVE AND TECHNICAL DIFFICULTIES OF USING AN EXPERIMENTAL DESIGN FOR EVALUATION WITH BROAD-AIM SOCIAL PROGRAMS ARE DISCUSSED, AND ELEMENTS OF A MORE EFFECTIVE EVALUATION DESIGN ARE SKETCHED.

Abstract

THE UNDERLYING ASSUMPTION BEHIND THE USE OF AN EXPERIMENTAL DESIGN IN EVALUATING SOCIAL PROGRAMS IS THAT SUCH PROGRAMS ARE DESIGNED TO ACHIEVE SPECIFIC ENDS AND THEIR SUCCESS CAN BE MEASURED BY THE EXTENT TO WHICH THESE ENDS WERE ACHIEVED. THIS ASSUMPTION AND CONSEQUENT USE OF EVALUATION METHODOLOGY ARE BELIEVED TO YIELD MISLEADING RESULTS WHEN THE ACTION-PROGRAMS HAVE BROAD AIMS AND UNSTANDARDIZED FORMS. BROAD-AIM PROGRAMS ARE CONCERNED PRIMARILY WITH IMPACTING ON A PROBLEM SITUATION AND ARE CONCERNED ONLY SECONDARILY WITH CHANGING INDIVIDUALS. AN EXAMPLE OF A PROBLEMATIC MISAPPLICATION OF AN EVALUATION EXPERIMENTAL DESIGN TO A BROAD-AIM PROGRAM IS DISCUSSED IN DETAIL. THE AIM OF THE PROGRAM WAS TO CHANGE EXISTING COMMUNITY INSTITUTIONS INCLUDING THE SOCIAL AGENCIES, THE SCHOOLS, AND THE EMPLOYMENT SERVICES OF A GIVEN DISTRICT TO EFFECT A POSITIVE CHANGE IN A DETRIMENTAL STATUS QUO. THE FOLLOWING GENERAL TECHNICAL PROBLEMS ARE ASSOCIATED WITH THE APPLICATION OF AN EXPERIMENTAL EVALUATION DESIGN TO THIS PROJECT: (1) THE PROBLEM OF DEVELOPING CRITERIA; (2) THE PROBLEM OF DEALING WITH AN UNCONTROLLED SOCIOLOGICAL CONTEXT FOR THE PROGRAM; (3) THE PROBLEM OF EVALUATING AN UNSTANDARDIZED TREATMENT; AND (4) THE PROBLEM OF USING AN EVALUATION DESIGN THAT DISCOURAGES UNANTICIPATED INFORMATION. ADMINISTRATIVE DIFFICULTIES INVOLVE THE FOLLOWING: (1) THERE MAY BE CONFLICT BETWEEN ADMINISTRATORS AND EVALUATION PERSONNEL OVER PROGRAM DEVELOPMENT; (2) AN OPERATION DESIGNED TO CONTRIBUTE TOWARD THE ACCOMPLISHMENT OF A BROAD GOAL IS OFTEN IDENTIFIED BY EVALUATORS AS A GOAL IN ITSELF BECAUSE OF THE DIFFICULTY OF EVALUATING THE MORE COMPLEX BROAD GOAL; AND (3) THE RESEARCH STAFF MAY KNOW LESS, RATHER THAN MORE, ABOUT THE CONSEQUENCES OF THE PROGRAM THAN THE ACTION GROUP KNOWS. WHILE THE PRESENTATION IS INTENDED PRIMARILY TO CITE THE FLAWS IN USING AN EXPERIMENTAL DESIGN WITH BROAD-AIM SOCIAL PROGRAMS, ELEMENTS OF A MORE EFFECTIVE METHODOLOGY ARE BRIEFLY PRESENTED. THE ELEMENTS DISCUSSED POINT TOWARD THE NEED FOR A MORE QUALITATIVE, PROCESS-ORIENTED APPROACH IN EVALUATING BROAD-AIM PROGRAMS. (RCB)