Background
Nationally, approximately 61 percent of children have been exposed to violence during the past year, and there is reason to believe that the problem has grown worse in recent decades (Finkelhor et al., 2009). Children's exposure to violence (CEV) can have serious consequences, including a variety of psychiatric disorders and behavioral problems, such as posttraumatic stress disorder (PTSD), depression, and anxiety. School performance has also been shown to suffer as a result. Moreover, research suggests that the effects of exposure to violence may persist well into adulthood (Margolin and Gordis, 2000). Though some research has shown that early childhood interventions can substantially improve children's chances of future social and psychological well-being, many such programs have not been evaluated to date, and still others have not been examined in real-world community settings. This article presents study data that were gathered in experimental and quasi-experimental studies conducted to attempt to fill these gaps in understanding about the effectiveness of programs intended to improve outcomes for children exposed to violence.
Safe Start Promising Approaches (SSPA) is one phase of a community-based initiative focused on developing and fielding interventions to prevent and reduce the impact of CEV. Sponsored by the U.S. Department of Justice's Office of Juvenile Justice and Delinquency Prevention (OJJDP), the Safe Start Initiative consists of four phases:
- Phase One: This phase focuses on expanding the system of care for children exposed to violence by conducting demonstration projects. Now complete, this phase involved demonstrations of various innovative promising practices in the system of care for children who have been exposed to violence.
- Phase Two: Building on Phase One, this phase was intended to implement and evaluate promising and evidence-based programs in community settings to identify how well programs worked in reducing and preventing the harmful effects of CEV on children.
- Phase Three: Still in the planning stages, the aim of this phase is to build a knowledge base of effective interventions in the field of CEV.
- Phase Four: The goal of this phase is to “seed” on a national scale the effective strategies identified in the earlier phases.
Each phase also includes an evaluation component that is intended to assess the implementation of the various interventions and their impact on children's outcomes.
This report focuses on Phase Two of the Safe Start Initiative. For this phase, known as SSPA, OJJDP selected 15 program sites across the country to implement a range of interventions for helping children and families cope with the effects of CEV. The 15 program sites varied in numerous ways, which are spelled out fully in a companion report, National Evaluation of Safe Start Promising Approaches: Assessing Program Implementation (Schultz et al., 2010). In short, the settings, populations served, intervention types, types of violence addressed, community partners, and program goals differed across sites. The RAND Corporation evaluated the performance of SSPA in each of the 15 program sites, in collaboration with the national evaluation team: OJJDP, the Safe Start Center, the Association for the Study and Development of Communities (ASDC), and the 15 program sites. The evaluation design involved three components: an outcomes evaluation; a process evaluation, including a program cost analysis; and an evaluation of program training efforts. This report presents the results of the outcomes evaluation.
Methods
The outcomes evaluations were designed to examine whether implementation of each Safe Start intervention was associated with individual-level changes in specific outcome domains. The evaluation utilized an intent-to-treat approach that is designed to inform policymakers about the types of outcomes that could be expected if a similar intervention were to be implemented in a similar setting. To prepare for the evaluation, the sites worked together with the national evaluation team to complete a “Green Light” process that developed the specific plans for the intervention and assured that the evaluation plan would align well with the intervention being offered and would be ethical and feasible to implement.
As a result of the Green Light process, a rigorous, controlled evaluation design was developed at each site, either with a randomized control group (wait list or alternative intervention) or a comparison group selected based on similar characteristics. Three sites had more than one study being conducted in their setting, as they delivered two different interventions to different groups of individuals. Overall, there were 18 separate evaluations of interventions within the 15 sites. Most sites utilized an experimental, randomized design (13 of 18), with RAND standardizing and monitoring the randomization procedures. Pre-intervention baseline data were collected on standardized, age-appropriate measures for all families enrolled in the studies. Longitudinal data on families were collected for within-site analysis of the impact of these programs on child outcomes at six, 12, 18, and 24 months post-enrollment. It should be noted that funding was relatively modest for designs of this type, requiring sites to leverage existing resources to the extent possible to maximize their ability to conduct the studies.
Measures for the national evaluation were chosen to document child and family outcomes in several domains: demographics, background and contextual, child and caregiver violence exposure, child behavior problems, child posttraumatic stress symptoms, child depressive symptoms, child social-emotional competence, parenting stress, caregiver-child relationship, and child school readiness/academic achievement. The 15 sites collected data with initial training and ongoing support from RAND and sent data to RAND for cleaning, scanning, and analysis.
Data analysis was performed for each site separately and included description of the baseline characteristics of families enrolled, description of the intervention services offered to these families, and mean responses on the outcome measures over time. The analysis of outcomes depended on the sample size, with more robust analyses and modeling possible with higher numbers of families in the samples.
Results
Descriptive analysis of the data collected confirmed the diversity of the interventions across the SSPA sites, with different types of families being served at the 15 sites. Across the sites, however, major impediments to the planned evaluations were the inability to recruit sufficient numbers of families for participation and difficulty retaining families for the follow-up assessments. Thus, none of the evaluations had adequate statistical power to detect whether the programs made an impact on participating children and families, and in some cases the samples were too low to complete exploratory analyses of changes within groups over time or differences between groups over time.
Within several sites, we noted promising improvements within the intervention groups over time. These include reductions in children's symptoms, improvement in resilience factors, and improvements in the parent-child relationship, depending on the site under study. In many cases, however, similar changes were noted in the control or comparison group, indicating that some outcomes may improve with the passage of time or by virtue of the general interventions received in the community.
Analyses of outcomes data in sites with marginally adequate statistical power to detect an intervention effect did not reveal any strong patterns of change in the outcomes in the intervention group as compared to controls. The report on the study presents possible reasons for these findings, including both methodological and programmatic reasons, and discusses next steps relevant to each site in terms of both research and practice.
Conclusions
As focus on CEV has increased among researchers and public agencies, and its negative consequences on health, behavior, and development have become more evident, intervention programs have been developed to try to improve outcomes for these children. However, many of these programs lack evidence of efficacy or effectiveness on child-level outcomes, and the few that have been empirically evaluated have not been well tested in community settings. As such, the evaluations conducted as part of the SSPA are important, as they attempt to rigorously examine the effectiveness of such programs delivered in community settings, even with relatively modest funding levels. The level of rigor in evaluation attempted here is rarely seen in implementation projects that operate under real-world conditions.
The diversity of the SSPA programs under study is also noteworthy. Although the programs all focused on children exposed to violence, they varied considerably in terms of the type of intervention, the setting in which it was offered, and the group of children and families targeted for the intervention. All of the programs were able to successfully launch and provide some services to children and families exposed to violence, as reported in our report on SSPA implementation (Schultz et al., 2010).
Obstacles to the successful evaluation of these programs were numerous, and some of them were at least partially surmounted. These included developing measures that could assess the sensitive topic of violence exposure and examine outcomes across a range of ages, using advanced psychometric techniques to develop measures that would span a broader age range, working with multiple institutional review boards to assure confidentiality of the data collected and define the limits to confidentiality, and engaging community partners and families into the evaluation studies. Other obstacles were harder to overcome, including limits in funding, low uptake of the intervention services in some cases, and struggles with recruitment and/or retention. All of the challenges weakened the ability to draw firm conclusions from the data collected. Despite the limitations, these data will be useful in planning future research endeavors. The difficulties faced in conducting this outcome evaluation will also provide useful information about the types of challenges faced by families entering intervention services and the challenges in evaluating child outcomes in a variety of settings. Practitioners using interventions such as these, or in these types of settings, are advised to examine this report and the process evaluation report (Schultz et al., 2010) to understand the nature of problems they might encounter and families' willingness to accept or adhere to services.
In summary, while the outcomes evaluation overall produced no conclusive findings about the effectiveness of the interventions under study, the SSPA process evaluation identified many successes of the individual programs in implementing their program goals (Schultz et al., 2010). These successes included development of procedures for increased identification of children exposed to violence, improving communication and coordination among service providers, and establishment of new interagency and communitywide partnerships to address service gaps for children and their families. These improvements have value in their own right and should be considered as part of the legacy of the Safe Start Initiative.
References
Finkelhor, D., H. Turner, R. Ormrod, and S. L. Hamby, “Violence, Abuse, and Crime Exposure in a National Sample of Children and Youth,” Pediatrics, Vol. 124, No. 5, 2009, pp. 1411–1423.
Margolin, G., and E. B. Gordis, “The Effects of Family and Community Violence on Children,” Annual Review of Psychology, Vol. 51, 2000, pp. 445–479.
Schultz, D., L. H. Jaycox, L. J. Hickman, A. Chandra, D. Barnes-Proby, J. Acosta, A. Beckman, T. Francois, and L. Honess-Morealle, National Evaluation of Safe Start Promising Approaches: Assessing Program Implementation, Santa Monica, Calif.: RAND Corporation, TR-750-DOJ, 2010. As of July 17, 2011:
http://www.rand.org/pubs/technical_reports/TR750.html
