Resource Using Evaluation Results to Improve and Sustain Sexual Assault Prevention Activities in the Military
This resource will help you process and use your evaluation data, which include a process evaluation (meaning how well the prevention activity ran relative to your plan) and an outcome evaluation (meaning what impact or changes resulted relative to your desired outcomes). See GTO Steps 7–9 in the GTO Handbook for details on how to summarize evaluation results and make quality improvement decisions.
Analyzing Different Kinds of Process Evaluation Data
- Prevention activity participant characteristics describe the demographics of the prevention activity participants (for example, number of participants, male or female, ethnicity, and age). This kind of information would likely come from a survey that was used before and after the prevention activity was run.
- Participants' utilization of prevention activity can be calculated from attendance information. You could calculate the percentage of participants who have perfect attendance (the number with perfect attendance divided by the number who participated), the overall attendance rate for the whole group (the total number of sessions attended by all divided by the total number of sessions that the group could have attended), or the overall attendance for each session of the prevention activity (the number of participants who attended the session divided by the total number of participants enrolled in the prevention activity). If the prevention activity consists of only one session, calculate attendance as a percentage of the total anticipated or targeted. Utilization could also mean exposure to a media campaign or the amount of online training completed. The main data point here is whether participants participated as expected.
- Level of delivery achieved by your prevention activity will depend on the measure you use. For example, you might calculate the percentage of activities fully completed, partially completed, and not at all completed for each session or component of a prevention activity.
- Participant satisfaction and implementers' perception of any type of prevention activity will also depend on the measure you are using. If you're using a measure that asks open-ended questions, look across the answers for general themes. If you are using a survey with defined answer choices, calculate averages or frequencies of the responses. NOTE: You might have collected information about beneficiaries of a prevention activity (rather than direct participants). However, the process for evaluating these data would be the same.
- Work plan adherence describes how closely your prevention activity plan was followed. The GTO Handbook includes a GTO Step 6 tool for outlining a prevention activity work plan. Each row in the work plan tool represents an action or milestone. Thus, the analysis of this data would be the percentage of actions or milestones that you left out (or completed) compared with those that you indicated would be done on your Step 6 work plan.
- Other could include any type of output of the prevention activity that might be important to track. For example, if the prevention activity required handing out materials to all individuals at an installation, what proportion of the individuals actually received those materials? Or if the prevention activity is a media campaign, track how much media was transmitted (emails, tweets, posters, etc.).
Analyzing Different Kinds of Outcome Data
Now that you have gathered your data, the next step involves analyzing it. It could be worthwhile to consult an expert in data analysis to ensure that you are using appropriate techniques. Just as there are quantitative and qualitative data collection methods, there are also quantitative and qualitative data analysis methods. When using quantitative data collection methods, such as surveys, it is common to use data analysis methods, such as comparing averages and frequencies. Sometimes your analysis could involve simply comparing your results on an outcome indicator to an established benchmark that you set for your prevention activity. If you are using evaluation measures from the prevention activity developers, then they might have scoring criteria or tell you what values are expected from prevention activity participants so that you can assess whether the prevention activity is having the intended effect. The Data Snapshot Tool for Prevention Activity Evaluation has a Microsoft Excel workbook that takes pre- and post-activity data entered into it for eight measures in the sexual assault prevention domain and automatically calculates averages for the data from before and after a prevention activity, displayed in bar chart form. Results are accompanied by brief summaries of the measures and how to interpret the chart. The Data Snapshot Tool is also available at www.rand.org/t/TLA746-1.
Interpreting and Reporting Evaluation Results
Whatever the results, you will need information from both your process and outcomes evaluation to tell you what is happening with your prevention activity. That is because, in order to reach desired outcomes, the prevention activity needs to be both implemented well and based on good evidence. In other words, good evidence + good implementation = results. A poorly designed prevention activity, even if implemented perfectly, will not produce desired outcomes. Conversely, a strong prevention activity that is implemented poorly will not produce desired outcomes. Therefore, if you do not achieve outcomes that you hoped for, a process evaluation can give you clues about why: If the process evaluation showed that the implementation was good, maybe the prevention activity is not ideal. If the process evaluation showed that the implementation was poor, maybe it was the poor implementation that led to poor outcomes. You can conclude this only with information from both types of evaluation. Interpreting your results in a thoughtful way helps you see what's working and what you need to change.
Finally, when interpreting outcome data, you will need to reconcile short-term outcomes with long-term outcomes. The evaluation of short-term outcomes could show that the prevention activity was successful (for example, service members who participated improved their knowledge of safe ways to intervene when they see a risky situation). However, it is possible that tracking long-term outcomes—for example, an installation's climate regarding sexual harassment—shows that the long-term outcome is unchanged. How can you reconcile those two results? One possibility is that not enough service members were exposed to the prevention activity to improve the long-term outcome. Another possibility could be that improving knowledge did not translate into actual behavior. Finally, it might be that not enough time has elapsed for the prevention activity to have an impact. As you can see from this example, long-term outcomes are more difficult to improve than short-term outcomes.
The conclusions that you draw from the data that you collect will help you develop a plan for continuous quality improvement (CQI), discussed in more detail in the GTO Handbook in Steps 7–9. For example, if the prevention activity seems to be working for those exposed to it, the improvement might simply be to increase the prevention activity delivery to more service members. Alternatively, maybe the prevention activity needs to be strengthened beyond targeting knowledge to include time to practice new skills, or perhaps additional prevention activities are needed to address other risk factors for sexual assault.
A word of caution: It is very possible that your evaluation will involve a small number of people (for example, 10–15), and the amount of change could be small. Therefore, you have to be careful in how much you can conclude from such an evaluation. Sometimes if there is a consistent pattern of evaluation results across multiple implementations of a prevention activity, you could be more confident in small changes in small samples. However, overall, it is wise to be cautious. See Using Getting To Outcomes to Plan Evaluation of Sexual Assault Prevention Activities for additional guidance. Of course, it could be very useful to consult an evaluation expert to help analyze and interpret the data.
Table 1 offers suggestions for tailoring evaluation results for different audiences.
Table 1. Evaluation Results and Reporting Methods Recommended for Specific Audiences
| Audience | Evaluation Results of Interest | Example of Reporting Method |
|---|---|---|
| Leadership |
|
Briefing; executive summary of findings; full report available upon request |
| Implementation team |
|
Detailed report with executive summary of findings |
| Service members, civilian personnel, and families |
|
Presentation, brochure, or webpage; all-hands meetings and town halls |
| Service headquarters |
|
Briefing; detailed report with executive summary of findings |
SOURCE: Adapted from Gordon Hannah, Sharon McCarthy, and Matthew Chinman, Getting To Outcomes in Services for Homeless Veterans: 10 Steps for Achieving Accountability, Philadelphia, Pa.: National Center on Homelessness Among Veterans, 2011. https://www.va.gov/homeless/nchav/docs/GTO_Homelessness_Manual.pdf
See also Engaging Leadership Before, During, and After Implementing Sexual Assault Prevention Activities in the Military for guidance on engaging and presenting results to leadership.
Sustaining Effective Sexual Assault Prevention Activities in the Military
The GTO Handbook contains important guidance for sustaining effective prevention activities. Your team can consider the following questions when planning for sustainability.
Table 2. How to Plan for Sustaining a Prevention Activity
What we have done in the past
- Does the need for the prevention activity continue? Has the need for the prevention activity changed or remained the same? Are there any new concerns?
- What particular result can we use to justify the prevention activity? Any goal or desired outcome that you achieved might be a good result to share with stakeholders to justify the prevention activity. Highlight any dramatic improvement from your data.
- What should we change about the way we do the prevention activity? Using your evaluation data and the CQI tool from Steps 7–9 in the GTO Handbook, think about the process—recruitment, enrollment, attendance, logistics, etc.—and consider whether one or more of these activities could be strengthened or changed to be a better fit for your site and staff.
- Who knows the prevention activity and supports keeping it going here? Consider which individuals at your site are champions of this prevention activity—that is, influential people who really like the prevention activity—and are enthusiastic about it, including leadership. Should somebody else be brought on? Who is going to take the lead?
What we will do in the future
- Where will the GTO tools, the prevention activity evaluation, and the prevention activity manual and materials be kept? Decide who will have access to them and consider how this is the same or different from other prevention activity materials at your installation. Where do you currently keep them? You need to make sure that completed tools from the GTO Handbook are accessible to all involved for future implementation.
- Who will be in charge of making the prevention activity happen? Also, think about who is trained to be the facilitator, how that decision was made, and what supervision would be necessary.
- Who else is in favor of the prevention activity and needs to be involved in keeping it going? Think about the different leadership levels and the community of service members.
- Who will do the evaluation and pre- and post-activity surveys, track attendance, and monitor adherence? When (how often) and to whom will the results be reported? Think about who could lead these activities (one person or more—staff or an outside group). Think about how to organize the results and who needs to see them.
- How much funding, if any, do we need for running the prevention activity? Are there resources other than funds that are needed to run the prevention activity well (for example, for recruitment, good attendance, supplies)?
- When will we run the prevention activity again? And when will we revise our work plan? Consider the different times of year or days and times of the week and what worked best in the past. Think about lead time needed to look back at the work plan from Step 6 in the GTO Handbook and revise it if needed.
- How can we keep staff trained in the prevention activity? The more staff are trained, the more likely you will be able to continue the prevention activity. Look back at the prevention activity materials and what is required of facilitators. Consider who could be trained and who would be responsible for doing the training.