Assessing Operation Purple

A Program Evaluation of a Summer Camp for Military Youth

by Anita Chandra, Sandraluz Lara-Cinisomo, Rachel M. Burns, Beth Ann Griffin

This Article

RAND Health Quarterly, 2012; 2(3):11

Abstract

Parental military deployments pose a host of challenges for child and family well-being. Military family support programs have proliferated since the start of the wars in Afghanistan and Iraq a decade ago to address these emotional, social, and academic issues, but there has been little evaluation of whether these programs are meeting their key objectives. To help fill this gap, a RAND study team explored the curriculum, themes, and outcomes of Operation Purple, a free weeklong summer camp program for youth with a deployed parent. Using a quasi-experimental approach, the study incorporated youth and parent survey data (from both camp attendees and a control group of non-attendees) and camp after-action reports and visitor observation logs to determine whether there were differences between attendees and non-attendees in the four camp theme areas: comfort and skill in communicating about feelings, understanding and appreciation of military life, sense of service/stewardship, and outdoor education. The study also sought to determine how and to what extent the program's curriculum was implemented by participating camps in the summer of 2011. Despite limitations in the data (e.g., a non-random study sample, some variation in curriculum implementation across camps), the study found some positive effects from camp participation, particularly in communicating about feelings, as well as valuable youth and parent perspectives about camp, reflected in responses to open-ended questions. As such, it helps lay the groundwork for future investigation of similar support programs for military youth and their families.

For more information, see RAND TR-1243-NMFA at https://www.rand.org/pubs/technical_reports/TR1243.html

Full Text

Over the past decade, research has highlighted the challenges that parental deployment can pose for the health and well-being of youth from military families. Cumulative months of parental deployment and associated stressors can have negative consequences for youth (De Pedro et al., 2011; Flake et al., 2009; Chandra, Lara-Cinisomo, et al., 2010; Chandra, Lara-Cinisomo, et al., 2011; Cozza, 2011) and for parents (Chartrand et al., 2008; Lara-Cinisomo et al., 2012). Although most military youth can navigate these experiences with little or no negative impact, these changes can cause distress among some youth. Studies from recent conflicts indicate that around one-third of children of deployed parents face higher levels of emotional difficulties and anxiety symptoms than youth in the general population (Flake et al., 2009; Chandra, Lara-Cinisomo, et al., 2010; Lester, Peterson, et al., 2010; Lester, Mogil, et al., 2011; Lester, Saltzman, et al., 2012; Chandra, Lara-Cinisomo, et al., 2011; Cozza, 2011). In addition, some military youth have reported challenges to the quality of peer and parent-child relationships (Huebner and Mancini, 2005) and academic problems, particularly those who have experienced 19 months or more of parental deployment (Richardson et al., 2011; Chandra, Martin, et al., 2010).

To address these emotional, social, and academic issues, the U.S. Department of Defense (DoD) and youth and family program leaders have developed a sizable number of programs intended to prevent or mitigate the impacts of parental deployment. Yet, there has been comparatively little effort to evaluate these programs. In light of recent research emphasizing priority issues for youth, the salient questions now are whether these programs are effective and whether existing programs are realizing their intended objectives. While there are several initiatives serving military youth, such as Operation Military Kids and Operation Purple® (the focus of this report), there has been very little independent evaluation of these programs. RAND has completed studies of Operation Purple camp applicants, but the focus to date has not been on program evaluation (Chandra, Burns, et al., 2008; Chandra, Burns et al., 2011).

This study sought to address this gap by evaluating Operation Purple, a summer program for military youth whose goals include helping youth cope with the stress associated with parental deployment. In 2004, the National Military Family Association (NMFA) began to address the need for support programs for this population, launching Operation Purple, a free, weeklong summer camp program for youth who have a deployed parent. During their stay, participants engage in a variety of fun activities while learning how to cope with the stress associated with the deployment of their parents. The curriculum for Operation Purple focuses on four themes, three of which (communication about feelings, military culture, and sense of service) focus on aspects of helping youth feel connected to the military and equipping them with the tools to cope with deployment-related stress. The fourth theme, engagement in outdoor activities, may have collateral benefits for youths' sense of calm.

Study Approach

The evaluation presented here had two objectives: (1) to assess whether Operation Purple is associated with self-reported improvements in the four main camp outcome areas (communication skills, understanding of military culture, sense of service, and outdoor education), which correspond to the four themes described earlier by comparing youth who attended the camp with those who applied but did not attend, and (2) to document how the 2011 curriculum was implemented by participating camps, using data from after-action reports (AARs) and visitor logs (VLs).

To address the first objective, the evaluation used a quasi-experimental design to track the effects of the camp on the four main outcome areas from the perspectives of participating children and adolescents (referred to as “youth” throughout this article) and their parents or primary caregivers (referred to as “parents” for simplicity) through a series of self-reported surveys administered before and at two time points after camp participation. While an experimental design would have been desirable for exploring causality and minimizing sample bias, randomization was not possible given the way that camp eligibility and acceptance were determined and NMFA's interest in retaining that approach (Rossi and Freeman, 1993; Bawden and Sonenstein, 1992). NMFA prioritizes acceptance for those youth who have an impending parental deployment and have not attended the camp in prior years. Thus, we identified a control group comprising youths who had applied but did not attend camp (because they either were not accepted or were accepted and did not attend) and created propensity score weights to adjust for the differences in baseline characteristics between the youth who attended an Operation Purple camp (the intervention, or camp, group) and those who did not (the comparison, or no-camp, group): applicant age, applicant gender, deployed parent's service and component (active versus National Guard or reserve), parental deployment status while the youth was at camp (or during the same period), number of deployments, camp attendance prior to 2011, receipt of any other (non–Operation Purple) military support or services during the study period, and all baseline survey items for outcomes of interest.

Data on the outcomes for the analysis of the first study objective (assessing the impact of camp participation) came from self-reported, web-based surveys conducted at three time points: during application (wave 1, or baseline), one month post-camp (wave 2), and three months post-camp (wave 3). The surveys were administered to both the youth and parent, with the same parent reporting in all waves. Fifty-seven percent of parents and 40 percent of youth who completed the baseline survey also completed the wave 2 survey. Fifty-five percent of parents and 50 percent of youth who completed the baseline survey also completed the wave 3 survey. These response rates are somewhat lower than those in other studies of similar populations (Chandra, Lara-Cinisomo, et al., 2011; Lester, Mogil, et al., 2011); however, this was a web-based survey, a method that generally has a lower response rate than other survey dissemination methods (median rate of 34 percent) (Bälter et al., 2005; Shih and Fan, 2008). After assessing the impact of camp on attendees, we carefully examined the effects on outcomes among first-time camp attendees using a smaller subset of camp applicants who had never attended an Operation Purple camp before 2011 to determine whether the program had a greater impact on this particular subsample of youth. Analyses also included qualitative data based on open-ended questions completed by respondents.

To assess our second study objective, we used data from Operation Purple AARs and VLs to measure how well the camps aligned with the program's themes and implemented required activities.

The AARs, completed by the camp directors, documented whether the four outcome areas were addressed, whether the required activities were carried out, and the activities that a given camp used to reinforce the four program components. These reports did not capture youth attendance at a given activity. The VLs provided information on the fidelity of implementation of two key activities: conservation and recycling (“Leave No Trace”) and military education and pride via the “Wall of Honor,” a collage that is created by camp participants to illustrate military pride. The VLs were completed by an outside, trained observer. Camp visitors (those completing the VL) were required to view a recorded webinar that reviewed expectations for their visit, the role they play in visiting camp, an overview of the camp curriculum and activities, and reporting procedures. The visitor selected the time and day of the observation but was encouraged to conduct the visit in the middle of the week to capture a more typical day. We did not assess inter-rater reliability because of a lack of project resources, and there were no quality assurance measures.

There was significant variation in the quality of information provided by camp directors in the AARs with regard to the extent to which activities were implemented. As a result, we relied on the information provided in the AARs to capture in a dichotomous way whether or not the required Operation Purple curriculum was implemented and, when possible, the frequency of specific activities related to the four required themes of the camp (communication skills, understanding of military culture, sense of service, and outdoor education). Our analyses relied on the VLs to provide a more objective measure of program fidelity, which involved a trained (outside) observer. However, while specific activities were observed during camp visits and mentioned in the VLs, the length of observation time spent by an observer was not reported in a standardized way, and the amount of information reported by the observers varied, making it difficult to reliably use that information.

Key Findings

Sample Characteristics

At baseline, we had 977 parent-child pairs. Of this set, 387 youth completed waves 1 and 2 of the survey, 491 youth completed at least waves 1 and 3, 560 parents completed waves 1 and 2 of the survey, and 542 parents completed at least waves 1 and 3.

Most of our sample came from Army families: Approximately 52 percent of the baseline sample came from the Army, 19 percent came from the Navy, 17 percent came from the Air Force, and 11 percent came from the Marine Corps. Approximately three-quarters (76 percent) of the sample came from the active component, with the rest from the reserve component (14 percent National Guard and 10 percent reserve). Fifty percent of the youth sample was female, with an average age of 11.0 years (standard deviation [SD] = 2.2). About 60 percent of the youth sample had experienced three or more parental deployments. Thirty-nine percent had attended Operation Purple before 2011, while the remaining group had never attended the camp before. For the purpose of our evaluation, 44 percent attended in 2011 (and served as our camp group); the remaining group formed the no-camp, or control, group. We accounted for camp attendance prior to 2011 in all of our analyses.

Communication About Feelings

One of the key principles of Operation Purple is that the camp will offer a safe and nurturing space for youth to discuss their feelings about parental deployment and military life and will provide youth with tools to explore those feelings thoughtfully, through journal writing or other expressive modes. On the one hand, there were no significant differences in communication comfort from the youth perspective between those who attended the camp in 2011 and those who did not. On the other hand, at the first follow-up assessment, parents whose children attended the camp in 2011 reported significantly greater improvement relative to parents in the no-camp group in terms of the youth's ability to make himself or herself feel better (38 percent of camp parents endorsed the item with higher confidence at wave 2 than at baseline versus 25 percent of no-camp parents), as well as in their sense of efficacy in helping their child feel better (27 percent of parents endorsed this item with higher confidence at wave 2 than at baseline in the camp group versus 15 percent of no-camp parents). At both follow-ups, parents in the camp group also reported greater improvement in interactions with the youth about how he or she was feeling (37 percent improved in the camp group versus 25 percent in the no-camp group at wave 2; 38 percent in the camp group versus 27 percent in the no-camp group at wave 3).

Results from the open-ended questions posed to camp participants and their parents one month after camp showed that a small number (31 out of 270 parent respondents and 14 out of 175 youth respondents) experienced some benefits associated with communication. For example, parents reported that their child was more willing and able to describe his or her feelings about being a military youth and about parental deployments.

Finally, results from the AARs revealed that, while all camps reported engaging youth in activities designed to promote the communication of feelings in verbal and nonverbal ways, only nine of 32 camps indicated that they conducted these activities on a daily basis. While camps were expected to include activities in the communication theme area, there was no expectation regarding how many activities were to offered and on how many days of camp.

Military Culture and Connection to Military Peers

Another core theme of Operation Purple camps is educating youth about military culture and fostering a sense of community in which military peers can connect with each other. Overall, there were no significant differences in comfort and understanding of military culture between youth who participated in the camp and those who did not; however, significantly more camp attendees reported having spoken with at least one servicemember outside their family at wave 2 (36 percent camp versus 21 percent no camp). There was a similar difference in terms of parents' reports that their child felt a sense of community. Twenty-seven percent of camp parents reported that their child felt a greater sense of community at the first follow-up relative to the baseline, compared with 16 percent of no-camp parents.

The findings from the qualitative data (i.e., open-ended questions) showed similar results. Thirty-one of 270 parents who responded to the open-ended question about the benefits of camp attendance reported that the camp helped their child become more familiar with military culture.

Among the four targeted themes, military culture was the least discussed in the AARs. Despite the amount of data available, the range of activities that reinforced this theme varied widely. The site visits targeted this area specifically. The results from our analysis of the VLs indicated that ten of the 28 observed sites did not fully implement the required activity.

Sense of Service/Stewardship

A third theme of the camp was to instill a sense of service among the youth and a commitment to community contribution, defined as helping peers or others. There was no significant difference between campers and non-campers with respect to self-reports of trying to help people in need and helping other military youth who may need help. This was true of reports from both youth and parents.

While there were no changes noted in the quantitative survey data, results from the qualitative data analysis (i.e., open-ended items included in the survey) indicated that, among those who responded to the open-ended questions, eight youth and 30 parents observed improvements in the youth's sense of stewardship as a result of camp attendance.

According to data extracted from the AARs, camps were able to promote the concept of stewardship, as defined by Operation Purple, through several practical and creative activities, such as writing letters to servicemembers. The VLs did not capture the stewardship component of the program.

Outdoor Education

A fourth theme was engaging youth in outdoor activities, which included education about the environment and related conservation topics, as well as a general appreciation of being outdoors. There were no significant differences between campers and non-campers in terms of appreciation of the outdoors.

Based on information gathered from the AARs, all camp directors reported at least one activity associated with Leave No Trace principles (or conservation and environmental preservation principles) and outdoor education. Results from the analyses using data extracted from the VLs support the variation reported in the AARs. However, observers noted that some camps either did not fully adhere to the required activity or simply did not implement the activity during the observation period.

Youth and Parent Reporting on the Benefits of Operation Purple Camps

Based on responses to the open-ended question posed to youth and parents in waves 2 and 3 about how Operation Purple camps help youth and parents, we found that there were potentially unintentional or secondary benefits to camp participation. Specifically, 25 percent of parents who responded to the open-ended question said that they observed that their child was more confident after attending an Operation Purple camp; fifteen youth also pointed to this secondary benefit in their responses. Other benefits included a sense of independence among participating youth. Specifically, 20 parents said that their child returned from camp behaving more independently than prior to camp attendance. The third benefit that emerged from the data was improved coping skills. Twenty percent of parents and 28 percent of youth said that the youth felt better equipped to cope with stress related to deployment as a result of camp participation. Finally, 20 parents said that Operation Purple camp gave their child an opportunity to take a break from the responsibilities and stress that comes with being a child of a parent in the military.

Concluding Observations

The evaluation of Operation Purple reveals that, from the parents' perspective, the program had some impact on youth comfort and ability to communicate about deployment-related stress. Given recent studies of military youth highlighting deployment-related stress and anxiety symptoms, this finding is important even though it was noted only by parents because being able to communicate about stress and feelings of anxiety is critical to addressing those symptoms (Stallard, Velleman, and Baldwin, 2001). Any signal of improvement in this area is important. For other camp areas, effects were not detected or were minimal.

These findings must account for the limitations of the study. Because we were unable to use random assignment, and because propensity score weights only control for differences between the camp and no-camp groups on observed characteristics, it is possible that unobserved baseline differences between our two groups may be biasing our results. For example, while we controlled for observed differences in baseline parent responses to the survey measures in our propensity score model, it may be that an unobserved key difference (such as level of engagement in the youth's life) may not be balanced between the camp and no-camp groups and that this difference, in turn, explains the finding derived from the parent responses. Informed by prior military family research (Chandra, Lara-Cinisomo, et al., 2011; Lester, Mogil, et al., 2011; Flake et al., 2009), we included many of the military and deployment characteristic variables, as well as camp participation factors, that may have affected the camp experience of 2011 participants and are associated with relevant outcomes, such as military connectedness and communication about stress. However, we did not have complete deployment history data, which have been critical in prior military family studies and could have been instrumental for propensity score weighting. We were also limited in our ability to construct a camp fidelity measure. While we attempted to use the AAR and VL data, the quality and validity of those data and the inability to conduct an independent assessment of camp implementation (based on resource constraints) hindered the extent to which we could incorporate these fidelity data in our analytic models. We used the AAR and VL data to contextualize our findings, but we acknowledge that a more complete implementation analysis would have strengthened our design. Finally, we note that youth who apply to camp (regardless of attendance) may already be distinct from other military youth, thus limiting our ability to generalize to the broader military youth population.

In spite of these limitations, this study fills an important gap in the military youth program evaluation research because it involved a rigorous quasi-experimental approach and used qualitative data to put findings into context and to highlight other potential benefits of programs like Operation Purple, which provide a safe and nurturing space for military youth to connect and share feelings about parental deployment. Our qualitative data showed that several youth and parents reported that campers experienced increased confidence and a sense of independence one month after camp (e.g., social and personal growth), items that were not explicitly measured in the quantitative surveys. Future studies should examine these other benefits. Further, our statistically significant findings associated with parents' responses suggest that there may be a “parental reprieve” effect (i.e., parents are better able to relate to youth after camp because they had a “break”); this should be examined in more detail. In addition, while this study focused on whether youth learned new skills in the four camp theme areas, additional research could examine whether this skill development translates to actual reductions in stress levels or anxiety symptoms related to parental deployment. Finally, analyses of camp implementation that include a more direct investigation of the type, frequency, and barriers to specific activity implementation could help explain why certain effects surfaced in this study and how camp processes could be improved to have a greater impact in the four camp theme areas.

The research described in this article was sponsored by the National Military Family Association and conducted jointly by RAND Health's Center for Military Health Policy Research and the Forces and Resources Policy Center of the RAND National Security Research Division.

References

Bälter, Katarina Augustsson, Olle Bälter, Elinor Fondell, and Yiva Trolle Lagerros, “Web-Based and Mailed Questionnaires: A Comparison of Response Rates and Compliance,” Epidemiology, Vol. 16, No. 4, July 2005, pp. 577–579.

Chandra, Anita, Rachel M. Burns, Terri Tanielian, and Lisa H. Jaycox, “Understanding the Impact of Deployment on Children and Families: Findings from a Pilot Study,” in Shelley MacDermid Wadsworth and David Riggs, eds., Risk and Resilience in U.S. Military Families, New York: Springer, 2011.

Chandra, Anita, Rachel M. Burns, Terri Tanielian, Lisa H. Jaycox, and Molly Scott, Understanding the Impact of Deployment on Children and Families: Findings from a Pilot Study of Operation Purple Camp Participants, Santa Monica, Calif.: RAND Corporation, WR-566, 2008. As of June 7, 2012: http://www.rand.org/pubs/working_papers/WR566.html

Chandra, Anita, Sandraluz Lara-Cinisomo, Lisa H. Jaycox, Terri Tanielian, Rachel M. Burns, Teague Ruder, and Bing Han, “Children on the Homefront: The Experiences of Children from Military Families,” Pediatrics, Vol. 125, No. 1, January 1, 2010, pp. 16–25.

Chandra, Anita, Sandraluz Lara-Cinisomo, Lisa H. Jaycox, Terri Tanielian, Bing Han, Rachel M. Burns, and Teague Ruder, Views from the Homefront: The Experiences of Youth and Spouses from Military Families, Santa Monica, Calif.: RAND Corporation, TR-913-NMFA, 2011. As of June 7, 2012: http://www.rand.org/pubs/technical_reports/TR913.html

Chandra, Anita, Laurie T. Martin, Stacy Ann Hawkins, and Amy Richardson, “The Impact of Parental Deployment on Child Social and Emotional Functioning: Perspectives of School Staff,” Journal of Adolescent Health, Vol. 46, No. 3, March 2010, pp. 218–223.

Chartrand, Molinda M., Deborah A. Frank, Laura F. White, and Timothy R. Shope, “Effect of Parents' Wartime Deployment on the Behavior of Young Children in Military Families,” Archives of Pediatric Adolescent Medicine, Vol. 162, No. 11, November 2008, pp. 1009–1014.

Cozza, Stephen J., “Meeting the Wartime Needs of Military Children and Adolescents,” in Jodef I. Ruzek, Paula P. Schnurr, Jennifer J. Vasterling, and Matthew J. Friedman, eds., Caring for Veterans with Deployment-Related Stress Disorders: Iraq, Afghanistan, and Beyond, Washington, D.C.: American Psychological Association, 2011, pp. 171–190.

De Pedro, Kris M. Tunac, Ron Avi Astor, Rami Benbenishty, Jose Estrada, Gabrielle R. Dejoie Smith, and Monica Esqueda, “The Children of Military Service Members: Challenges, Supports, and Future Educational Research,” Review of Educational Research, Vol. 81, No. 4, December 2011, pp. 566–618.

Flake, Eric M., Beth Ellen Davis, Patti L. Johnson, and Laura S. Middleton, “The Psychosocial Effects of Deployment on Military Children,” Journal of Developmental and Behavioral Pediatrics, Vol. 30, No. 4, August 2009, pp. 271–278.

Huebner, Angela J., and Jay A. Mancini, Adjustments Among Adolescents in Military Families When a Parent Is Deployed: Final Report to the Military Family Research Institute and Department of Defense Quality of Life Office, Falls Church, Va.: Department of Human Development, Virginia Tech, June 2005.

Lara-Cinisomo, Sandraluz, Anita Chandra, Rachel M. Burns, Lisa H. Jaycox, Terri Tanielian, Teague Ruder, and Bing Han, “A Mixed-Method Approach to Understanding the Experiences of Non-Deployed Military Caregivers,” Maternal and Child Health Journal, Vol. 16, No. 2, February 2012, pp. 374–384.

Lester, Patricia, Catherine Mogil, William Saltzman, Kirsten Woodward, William Nash, Gregory Leskin, Brenda Bursch, Sara Green, Robert Pynoos, and William Beardslee, “FOCUS (Families OverComing Under Stress): Implementing Family-Centered Prevention for Military Families Facing Wartime Deployments and Combat Operational Stress,” Military Medicine, Vol. 176, No. 1, January 2011, pp. 19–25.

Lester, Patricia, Kris Peterson, James Reeves, Larry Knauss, Dorie Glover, Catherine Mogil, Naihua Duan, William Saltzman, Robert Pynoos, Katherine Wilt, and William Beardslee, “The Long War and Parental Combat Deployment: Effects on Military Children and At-Home Spouses,” Journal of the American Academy of Child and Adolescent Psychiatry, Vol. 49, No. 4, April 2010, pp. 310–320.

Lester, Patricia, William R. Saltzman, Kirsten Woodward, Dorie Glover, Gregory A. Leskin, Brenda Bursch, Robert Pynoos, and William Beardslee, “Evaluation of a Family-Centered Prevention Intervention for Military Children and Families Facing Wartime Deployments,” American Journal of Public Health, Vol. 102, No. S1, March 2012, pp. S48–S54.

Richardson, Amy, Anita Chandra, Laurie T. Martin, Claude Messan Setodji, Bryan W. Hallmark, Nancy F. Campbell, Stacy Hawkins, and Patrick Grady, Effects of Soldiers' Deployment on Children's Academic Performance and Behavioral Health, Santa Monica, Calif.: RAND Corporation, MG-1095-A, 2011. As of June 7, 2012: http://www.rand.org/pubs/monographs/MG1095.html

Shih, Tse Hua, and Xitao Fan, “Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis,” Field Methods, Vol. 20, No. 3, August 2008, pp. 249–271.

Stallard, Paul, Richard Velleman, and Sarah Baldwin, “Recovery from Post-Traumatic Stress Disorder in Children Following Road Traffic Accidents: The Role of Talking and Feeling Understood,” Journal of Community and Applied Social Psychology, Vol. 11, No. 1, January–February 2001, pp. 37–41.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.