Evaluate the Implementation Process

Before describing approaches to planning your process evaluation, we first provide an overview of some of the different types of evaluation that may be most relevant to your project. The term "evaluation" refers to the systematic analysis of data and information related to your work to help you achieve better outcomes for your program and families. Evaluation is a central tool of effective program planning, implementation, and ongoing management, and GTO uses evaluation—mostly in Steps 7 and 8 but also throughout each of the 10 steps—as a way to help you achieve the outcomes you seek.

Types of evaluations

The unit or level of evaluation can vary, and in the case of your program, you are probably interested in evaluation at several levels. The first is population-level evaluation, which refers to analysis of the entire community or the other geographic units that you serve. Your needs assessment in Step 1 is an example of a population-level evaluation. The second level of evaluation that you will want to consider is program-level evaluation, in which you analyze the performance of your program, addressing such issues as whether you are in fact serving your target population and whether you are implementing the home visiting model you selected with fidelity. A third level of evaluation is an outcome- or individual-level evaluation, where you analyze information on the families you serve. This might take the form of examining how long families stay in your program or whether they improve consistent with the goals of the program.

You can find dozens of different types of evaluations described in numerous textbooks and other references. For more detailed information about practical program evaluation approaches, see Chen (2005). While it is not necessary to engage in every type of evaluation for your program, you will want to conduct some evaluation at each of the three levels just outlined. You have already conducted a population-level needs assessment in Step 1, and you will revisit population-level evaluation in Step 8.

In this step, we describe a specific kind of program-level evaluation called process evaluation. In Step 8, we also discuss how to use individual-level outcome evaluation—or the program's impact on individuals—to inform your work. The process and individual-level evaluations together tell the story of your program, and, therefore, GTO-HV recommends doing both. We describe process evaluation in this chapter, Step 7, and individual-level evaluation in Step 8.

Overview of process evaluation

At this point, you have selected your home visiting program and planned how to roll it out. Before you actually implement your program, it will be important to spend some time planning what is called a process evaluation, in which you analyze information at the program level. A process evaluation monitors and documents the program's implementation processes throughout the life of the program.

Process evaluations are designed to give you an idea about how well the program plans were put into action. This can include assessing whether the amount and frequency of services you planned have actually been delivered (i.e., dosage), whether the people who participated in the program have been satisfied with their experience, and the degree to which the program has been carried out with fidelity, or faithfulness to the design of the program developers. Assessing dose, satisfaction, and fidelity can quickly give you important places to make mid-course corrections that will help improve a program's operation.

In the next step (Step 8), you'll work on planning an outcome or individual-level evaluation, which will help you to document the outcomes (for example, reductions in child maltreatment). We encourage you to read and work through the planning portions of both Steps 7 and 8 before launching your home visiting program.

Before implementation: Planning to evaluate the process

A major part of planning your process evaluation is identifying which program elements (or "components" in GTO-HV language) you need to monitor. In addition, a process evaluation plan should be able to answer the following questions:

• What process evaluation questions should we ask?
• What tools should we use?
• What should our data-gathering schedule be?
• Who is responsible for gathering the information?

Once it is begun, the results of a process evaluation can provide answers to the following questions:

• What are the characteristics of the families who received the program?
• How many families participated?
• How often did families participate?
• How did families hear about the program?
• Were the program activities implemented with fidelity?
• How satisfied were the participants with the activities?
• What do home visitors and other staff think of the program delivery?

For example, if you planned a program that involves three home visiting sessions to teach parenting and other skills, your process evaluation activities would probably include all of the following:

• Collecting participant characteristics
• Tracking the number of home visits received by each family to see whether most families received all three sessions
• Collecting information from the home visitors to make sure that they covered all the session material during the home visits
• Conducting satisfaction surveys with families during and/or after the program to see what they thought of the program
• Surveying home visitors to see whether they feel that the participants were engaged.

Elements of data collection

There are many approaches to conducting a process evaluation. The program model you select may require that you collect additional data elements, but we highly recommend that you collect the following types of process evaluation data:

• Participation—Keep track of each participant over time by creating a roster and list the dates of each session for each participant. You may also want to keep track of some characteristics about your participants, such as age, gender, marital status, and referral source. These data should be kept confidential.
• Fidelity monitoring—Check to see that the program is being delivered as intended. This includes determining whether home visitors received the appropriate training and technical support, whether participants are receiving the appropriate number of sessions specified by the program, and whether home visitors are touching on all of the required topics during their visits. If you are using an evidence-based program, fidelity measures have likely already been created. Check with the program representative—there may be certain data elements that you are required to collect in order to be allowed to implement the program.
• Monitor your work plan—You should be following your work plan as you implement your program. You can use the work plan you created in Step 6 to track the completion of your activities.

If you have the resources and time, we suggest you also collect the following information on your program's implementation:

• Participant satisfaction—Participants' perceptions of your program can be collected using brief surveys.
• Feedback through focus groups—Focus groups are a good way to solicit feedback on program satisfaction and gather suggestions for improvement. Focus group participants might include participating families, staff of referring organizations, the home visitors, or other individuals.
• Staff perceptions—Solicit ideas from your home visitors, supervisors, and other staff on perceived successes and challenges in implementing your program. This will help you to identify which factors facilitated the program's implementation and which factors may have emerged as barriers. The information can be tracked over time to determine whether the barriers identified have been adequately addressed.

Note that technology such as computers in the form of portable tablets can be very helpful in streamlining data collection and survey responses, especially if staff are stretched thin.

We have assembled process evaluation data-gathering tips to help you think through ways to gather the types of process evaluation information that we noted above.

Develop a Process Evaluation Plan

The Process Evaluation Planning Tool in this section can help you organize your process evaluation plan. The tool has two worksheets, the Process Evaluation Planning and Plan for Making Mid-Course Corrections.

Ready to implement: Conduct a process evaluation and analyze the results

Once you have developed your plan, you'll be ready to conduct your process evaluation. This will happen during your program implementation. For example, you will be gathering information about the target populations' characteristics, the amount and fidelity of the program delivery, and the perceptions about the program from the participants and staff. Once you have the process evaluation data, you should be ready to make mid-course corrections if necessary using the plan for mid-course corrections.

We recommend—again—that you take time to read through Step 8 (and the rest of the manual) before you implement your program. This will help you consider whether you need to collect information from or about program participants before participants receive the home visiting program.

Before moving on to Step 8

Once you've finished your process evaluation plan, you are ready to move on to Step 8, in which you'll plan your outcome evaluation to examine whether you are achieving the changes you seek for individuals and your community.

• ← Previous Chapter
• Next Chapter →