Approaches to Evaluation

This section includes a brief summary of both the types of evaluation and the components of evaluation. In addition to the information provided here we provide links to external sources of information on evaluation that may be of use.

Types of Evaluation

Planning Evaluation

Evaluation planning should begin at the earliest stages of the project. Not only is it a useful exercise for clarifying the goals of the project but also it allows one to identify the appropriate points in the project where evaluation can take place. Finding the appropriate evaluation points in the design and timetables of the project will both capture the most information and also will mitigate the side effects of the evaluation. For example, if certain scheduled events are taking place, evaluation efforts can "piggy-back" on those events rather than have an evaluation activity (e.g., survey) as a separate event. The key to mitigating negative effects is to be as non-obtrusive as possible in data collection methodology.

This type of evaluation is used to clarify the goals of the project in preparation for project implementation and also to create specific statements of goals, defined in measurement terms to base an evaluation.

• Budgeting for Evaluation
  "A general guideline is to allocate 5-10 percent of project costs for the evaluation of large-scale projects (those exceeding $100,000); for smaller projects, the percentage may need to be higher to meet minimum costs of fielding evaluation activities."[(2), page 18]

Formative Evaluation
A formative evaluation provides information about a project as the original plan is put into place and during the initial stages of the project's life where early changes could have large impact on the success of the project. Typically, a Formative Evaluation evaluates the process and any intermediate goals.

Formative Evaluation has two components:

• Implementation Evaluation provides information about the project as it is initially being implemented. By implemented we do not include the design of the project. During the implementation one is simply implementing the plan that was developed prior to the start of the project. Thus an implementation evaluation evaluates how a project was initially implemented. A typical question posed for this type of evaluation "Did the series of activities/interventions follow the protocol described in the project design statement?"
• Progress Evaluation provides information about the project during the early stages of running. Its primary use is in helping guide the project towards its goals during the life of the project. The primary purpose of this type of evaluation is to assess progress in meeting program goals. Typical questions posed in this type of evaluation: "Are participants achieving intermediate goals?" "Are participants improving/progressing at a rate consistent with achieving the final goal?"

Summative Evaluation
Summative evaluations occur just prior to or just following the completion of a project. They assess the success of a project in terms of its goals and perhaps even in terms of its unanticipated outcomes. The key to summative evaluation is to evaluate project outcomes (i.e., Were project goals achieved?)

Major Evaluation Components

Question Development
The development of questions begins with a solid understanding of the goals and anticipated outcomes of the project. Developing the goals and outcomes of the project is non-trivial and may be the most time consuming part of question development. Following the development of the project goals one typically develops questions around them with the goals of the evaluation in mind. Questions need to be properly reviewed to ensuring the wording and the scaling of the question (e.g., Likert scale) are appropriate for measuring project outcomes. Professional standards in measurement and in survey methodology must be followed.

Collection Method Identification
Once questions have been developed there are a variety of methods that can be employed to gather information from specific information holders. Common methods include surveys, focus groups, interviews, etc. Different data collection methods can be more efficient in covering breath of information (e.g., survey) or in obtaining more depth in information (e.g., interviews.)

Data Analysis
Just prior to analysis, data need to be cleaned and checked to eliminate potential problems such as data coding errors. Initially simple summaries of the information employing descriptive statistical analyses (e.g., mean. standard deviation) will be reported. More sophisticated data analytical techniques may be needed depending upon the goals of the project.

Report Generation & Dissemination
Following the analysis of the data collected useful and interesting presentations of the data can be developed and disseminated to project stakeholders. Project stakeholders typically include the Principal Investigator (PI), the project members and funding agency. Naturally different subsets of the information may be more interesting to one stakeholder than another and one may consider creating separate reports for each stakeholder. The most important consideration for reports is that they clearly outline project goals, evaluation techniques used, the results, and a discussion. Ultimately the findings need to be utilized by the appropriate stakeholder groups.

(1) The 2002 User-Friendly Handbook for Project Evaluation

(2) User-Friendly Handbook for Project Evaluation