Resource Using Getting To Outcomes to Plan Evaluation of Sexual Assault Prevention Activities

This resource will help you design your prevention activity evaluation, which includes a process evaluation (meaning how well the prevention activity ran relative to your plan) and an outcome evaluation (meaning what impact or changes resulted relative to your desired outcomes) - see GTO Step 6. Table 1 illustrates the difference between process and outcome evaluation. It is best to plan the content for the process and outcome evaluations before you launch your prevention activity. This will allow you to more effectively monitor your prevention activity while it is running. The remainder of this resource offers guidance on how to conduct process and outcome evaluations. Another resource (Process and Outcome Evaluation Measures for Sexual Assault Prevention Activities in the Military) includes examples of process and outcome measures for sexual assault prevention activities.

Process Evaluation	Outcomes Evaluation
tracks quality of a program Who participates (Did we reach the right target population)? How much do participants get (Did target population get right "dose")? How well was the program delivered (adhered to curriculum)? How satisfied are participants?	tracks change in participants How much did participants change on the outcome of interest (actual behaviors or behavioral risk and protective factors)?

What Is a Process Evaluation?

Process evaluation involves evaluating the implementation of a prevention activity: Did it run according to your plan, and how well did it go? This step is called process evaluation because the collected data track the process of prevention activity implementation, as opposed to the outcomes experienced by the participants. Process evaluations typically track attendance of participants, prevention activity adherence, and how well you followed your work plan. They could also involve asking prevention activity participants or implementers about how well they thought the prevention activity was delivered. A variety of methods for process evaluation are presented in Process Evaluation Methods, below. A process evaluation should be planned before a prevention activity begins and should continue while it is running.

Note about other prevention activities: Even though this process evaluation guidance is focused mostly on programs, much of the information can be used for conducting an evaluation of other prevention activities, such as changes in site policy. In that case, you will want to find out the extent to which service members know about the policy and the consistency of its implementation.

Why Is Process Evaluation Important?

The process evaluation tells you how well plans are being put into action and helps you routinely and systematically monitor areas that are important to making a prevention activity (including policy change) successful. Example questions include the following:

• Compared with your expectations, how many people attended or took part in the prevention activity (for example, how many were exposed to the new policy)?
• How much of the prevention activity did participants receive? What was the dropout rate?
• Did the facilitator deliver the entire prevention activity as designed? (How consistently implemented was the new policy?)
• Were participants satisfied with the activity?
• What were the perceptions of those who implemented the prevention activity?

The process evaluation will also tell you whether you need to make midcourse corrections (for example, whether you should improve attendance because attendance was weak) or changes to your work plan for your next round of implementation. Such data will provide you with information that could be useful to other sites doing the same prevention activity and to planners in military leadership and could help you better understand your prevention activity outcomes.

Process Evaluation Methods

You are likely to use a variety of methods for collecting your process evaluation data. Here's some additional information about a few key methods mentioned in this resource.

Source: Adapted from Hannah, McCarthy, and Chinman, 2011.

What Is an Outcome Evaluation?

An outcome evaluation reveals how well a prevention activity met the goals and desired outcomes set for it in GTO Step 2. Did the participants change on the desired outcomes, such as knowledge, attitudes, and behaviors? This step is called outcome evaluation because the collected data track the desired outcomes of the prevention activity (established in GTO Step 2), as opposed to the process of implementation described above. Each prevention activity outcome evaluation should be planned before the prevention activity begins and should have specific time points for data collection, such as before and after a complete cycle. Each of your prevention activities should have an outcome evaluation plan.

Why Is an Outcome Evaluation Important?

The purpose of the outcome evaluation is to understand whether you have met your desired outcomes for each prevention activity. Combined with the results of your process evaluation, this step will begin identifying areas for improvement to help address any missed outcomes in an effort to improve the prevention activity while maintaining achieved outcomes. Outcome evaluation results can help you demonstrate the effectiveness of a prevention activity to military leaders and other stakeholders.

How Do I Do an Outcome Evaluation?

For each prevention activity, you need an outcome evaluation design and a data collection and analysis plan, including a measurement tool (for example, a pre-survey and a post-survey), a target population to be measured (for example, all the participants in the prevention activity), a timeline for when to collect the data (for example, before and after implementation), a plan for entering the collected data (usually into a spreadsheet), and a plan for analysis to determine whether outcomes were achieved (for example, the change from the pre-survey to the post-survey). Outcome evaluations can be complex, costly, and intimidating. This resource is meant to assist with simple outcome evaluations. If you want to carry out more-complicated outcome evaluations, you might need to get help from a trained prevention activity evaluator.

Planning the Outcome Evaluation Design

Design refers to the type of evaluation you will conduct. The type of design guides when you collect data and from which groups. For example, a simple and inexpensive design uses a questionnaire to collect data from prevention activity participants just before a prevention activity begins and after it is completed (often called a pre-/post-). This design might be appropriate to assess changes in knowledge and attitudes that were targeted by a prevention activity. Another type of design, called the pre-/post- with comparison group, compares participants with a similar group not receiving the prevention activity during the same time period. This way, you can be sure that any changes taking place in the participants receiving the prevention activity from pre-program to post-program were real and did not also happen to nonparticipants (for example, if both groups improve the same amount, then the prevention activity did not have an effect). This design improves confidence that differences were due to the prevention activity and not to something else. That is why this design is a stronger way to evaluate whether the prevention activity led to changes in knowledge, attitudes, or behaviors over time. However, this design is more complicated, so you might want to consult an evaluator for assistance.

Doing a pre-program assessment and then a post-program assessment involves two surveys. There are instances in which it might be better to survey once to collect both pre-program and post-program results after the program. How can you get a pre-program assessment after participants have gone through the program?

In a technique called the retrospective pre-/post-, you ask people two sets of questions in the same post-program assessment:

1. Pre-program questions: Ask people to think back to before they took the program to answer the question (for example, "Before participating in program X, what did you know about . . . ?").
2. Post-program questions: Use the same process as any post-program assessment.

You would then analyze the data just like you would a regular pre-/post- assessment (comparing pre-program and post-program scores). You could even use the Data Snapshot Tool (described below) to analyze the data.

This technique has multiple benefits. First, it is much easier to do than conducting two assessments. You have to administer the survey only one time, and it is much easier to link a person's pre-program data to their post-program data. Second, it might actually be more accurate. There are some instances (especially when assessing knowledge or attitudes, less so with concrete behaviors) in which people might overestimate how much they know (or believe) about a topic on the pre-program questions, but, after they go through the program, they learn that they did not know (or believe) as much as they thought and might actually give a lower score on the post-program questions. Thus, it might look like they did worse over time, even though they might actually have improved. Using this technique could be a good evaluation option. The drawbacks are that the post-program survey takes longer, the retrospective items have to be reformatted, and you have to put in instructions to guide respondents to think back to the pre-program time period.

Planning For Collecting Data On Your Outcomes

There are many methods you could use to collect data on your outcomes. Table 2 provides an overview of pros, cons, and costs of many of the common methods. A common method is an outcome survey conducted before and after the prevention activity implementation. It measures outcome domains with individual survey questions or several questions grouped together into topical categories called scales (see Process and Outcome Evaluation Measures for Sexual Assault Prevention Activities in the Military for examples of relevant measures). For example, a measure assessing knowledge of workplace harassment might include several questions assessing different types of harassment knowledge. The question responses can be averaged together to form a single score, or scale. Then, the analysis of these data can also be done easily by scoring each scale, calculating the average for the group surveyed, and then comparing the pre-program and post-program scale scores. These surveys also can use individual items to assess change in knowledge, attitudes, and behaviors.

Choosing, Entering, and Analyzing Your Outcome Data

If you are using GTO, in Step 2 you specified measurable desired outcomes. Planning an outcome evaluation includes deciding what measures you will use. It can be advantageous to use measures that already exist rather than creating your own. For example, there are several well-known and tested measures of the outcomes, including ones used in the military. See Process and Outcome Evaluation Measures for Sexual Assault Prevention Activities in the Military for a list of measures and information about them.

One tool that can help you choose measures and enter and analyze your data is the Data Snapshot Tool for Prevention Activity Evaluation, which was created to support Department of Defense efforts to carry out basic evaluations for programs related to sexual assault and harassment prevention. The Data Snapshot Tool is also available at www.rand.org/t/TLA746-1. (If you are a military practitioner, be sure to contact your service for any additional guidance before using this tool.)

The Data Snapshot Tool components are as follows:

1. The Data Snapshot Tool Instructions (Word document) provide information on eight survey measures that can be used to estimate the impact of prevention efforts, details about facilitating surveys, and guidance on data collection and analysis using the tool.
2. The Data Snapshot Tool (Excel file) is a blank, preprogrammed Excel workbook that allows for easy data entry and analysis of results from the provided survey measures.
3. The Data Snapshot Tool - Usage Example (Excel file) is a completed version of the Data Snapshot Tool showing data entry and results pages.

The Data Snapshot Tool Instructions provide details about the four steps involved in executing a program evaluation:

Step 1: Select Survey Measures. The instructions contain details and actual survey items from eight measures that assess outcomes relevant for sexual assault and harassment prevention.

Step 2: Administer a Survey. The instructions contain details on how to administer surveys, including tools that facilitate proctoring a survey.

Step 3: Enter Data. The instructions provide detail on how to enter data into the Excel workbook from surveys conducted before and after a prevention program is run (see Figure 1).

Step 4: Analyze Data. The Microsoft Excel workbook automatically calculates averages and displays the data from before and after a prevention program in bar chart form. Results are accompanied by brief summaries of the measures and how to interpret the chart.

It is very possible that your evaluation will involve a small number of people (for example, 10–15), and the amount of change could be small. Therefore, you have to be careful about how much you can conclude from such an evaluation. One way to boost your confidence that small changes in small samples are meaningful is looking for patterns across multiple implementations of a program. For example, if the prevention activity is implemented with ten different groups of service members and the evaluation results, while showing a small positive change, consistently show the same small change in each of the ten groups, you could be somewhat more confident that the results were genuine than if you had the evaluation results of just one group. Another strategy for boosting confidence is by seeing whether the results across all the survey items follow the pattern you would have predicted. For example, say that you have a program that strongly emphasizes bystander intervention skills and touches only on rape myths. In the evaluation, respondents are asked items that gauge change on bystander skills and endorsement of rape myths. One could predict that the participants would show more change on the bystander skills than on the endorsement of rape myths. If those results come to pass, then one could have a little more confidence in the results, even if they were small changes (and especially if those results were repeated across several groups).

Also, it is important to try to understand the data before presenting them to others who are not directly involved in the evaluation. Again, consulting an evaluation expert could be useful in interpreting the results.

Reference

• Hannah, Gordon, Sharon McCarthy, and Matthew Chinman, Getting To Outcomes in Services for Homeless Veterans: 10 Steps for Achieving Accountability, Philadelphia, Pa.: National Center on Homelessness Among Veterans, 2011. As of November 1, 2021: https://www.va.gov/homeless/nchav/docs/GTO_Homelessness_Manual.pdf

Handbook Overview