Aug 25, 2011
RAND conducted the national evaluation of Safe Start Promising Approaches (SSPA) &mdeash; an initiative that involved 15 sites implementing promising and evidence-based children's exposure to violence (CEV) programs in community settings — to identify how well such programs work in reducing and preventing CEV's harmful effects. Overall, because of both methodological and programmatic issues, the outcomes evaluation revealed many inconclusive findings about the impact of interventions on child-level outcomes. However, the SSPA process evaluation identified many successes among the individual programs in implementing program goals, including developing procedures for increased CEV identification, improving communication and coordination among service providers, and establishing new interagency and communitywide partnerships to address service gaps for children and their families. These improvements have value in their own right and should be considered part of the legacy of the Safe Start Initiative.
In recent years, approximately 61 percent of children in the United States reported that they were exposed to violence, crime, and abuse during the previous year. Common sources of children's exposure to violence (CEV) are direct child maltreatment, witnessing domestic violence, and community and school violence. There is reason to believe the problem may have grown worse in recent decades. Child protective services agencies accepted 3.3 million referrals for neglect and abuse in 2008, and, from these referrals, 772,000 children were found to be victims of abuse or neglect. This figure represents a national rate of 10.3 victims per 1,000 children. Also, a high proportion of children are exposed to violence that occurs between adult intimate partners. Finally, exposure to violence outside the home is also common, with youth being victims of violent crime at a rate of 26.5 per 1,000 in 2009.
CEV also represents a risk factor for a variety of problems, including psychiatric disorders and behavioral problems, such as post-traumatic stress disorder (PTSD), depression, anxiety, and developmental problems. School performance has also been shown to suffer as a result of CEV. Research suggests that the effects of exposure to violence may persist well into adulthood.
Given the risk to children's health and well-being from CEV, there is a real need for effective prevention and intervention programs. But, despite some progress in identifying evidence-based models and practices, programs to date are limited in their reach (e.g., are limited to particular age ranges, settings, or symptoms), and few have been well tested in community settings. Demonstration projects have rarely moved beyond the description phase and into rigorous test interventions and impact evaluations. This means that community practitioners have little research to draw on when attempting to address the needs of the trauma-exposed families with whom they work.
The need both to develop better CEV programs and practices and to demonstrate that they can work in community settings was the impetus for the Safe Start Initiative launched by the Office of Juvenile Justice and Delinquency Prevention (OJJDP) in 2000 as a community-based initiative focused on developing, fielding, and evaluating interventions to prevent and reduce the impact of CEV. The initiative consists of four phases, including a first phase (completed in 2006) involving demonstrations of various innovative, CEV-related promising practices in the system of care.
The RAND Corporation was selected to conduct the national evaluation of the second phase — the implementation of 15 Safe Start Promising Approaches (SSPA) CEV programs in community settings — to identify how well such programs work in reducing and preventing CEV's harmful effects. This research highlight summarizes the results of the process and outcomes evaluations published in two RAND reports.
In 2005, OJJDP selected 15 SSPA program sites spanning cities and counties across the country: the Bronx, New York; Broward County, Florida; Chelsea, Massachusetts; Dallas, Texas; Dayton, Ohio; Erie, Pennsylvania; Kalamazoo, Michi- gan; Miami, Florida; Multnomah County, Oregon; Oakland, California; Providence, Rhode Island; San Diego County, California; San Mateo County, California; Toledo, Ohio; and Washington Heights/Inwood, New York. The sites drew from the demonstrations of innovative promising approaches in the first phase and were selected based on the strength of the evidence and theory base of their designs and on the feasibility of the interventions.
The 15 sites were also diverse in terms of community size, populations served, types of violence exposure, intervention strategies, and implementation settings. For example, community sizes ranged from those with catchment areas with fewer than 500,000 residents to those with more than 1 million residents. The sites were also racially and ethnically diverse, ranging from 85 percent white in Kalamazoo to 36 percent black in Oakland to 57 percent Hispanic in Miami.
Within their communities, the SSPA programs were situated in or overseen by a variety of different lead agencies or organizations, ranging from health clinics or hospitals to domestic violence agencies to human services agencies. SSPA sites varied in terms of the age range of children served: Most served families with young children, but some also served families with children up to 18 years old. Targeted populations were also diverse, ranging from children exposed to domestic violence or other forms of violence to children exiting domestic violence shelters to children who were victims of abuse and neglect. The referral agencies themselves were also diverse, ranging from clinics and hospitals to domestic violence shelters to Head Start classrooms.
To understand and learn from the variety of interventions in community settings, researchers grounded their evaluations in a conceptual model (shown in the figure) that was adapted from other research that involved evaluating the implementation of community-based mental health programs. In this model, a planned program includes such components as program design and target population. Planned implementation support includes preplanning activities, the quality of program materials, technical support, and implementer readiness. Both the planned program and the planned implementation support feed into the program as it is actually implemented.
|Conceptual Framework for Evaluating SSPA Programs|
The program as it is implemented is also influenced by other factors: community, provider, intervention, delivery system, and support system characteristics. The researchers organized these factors into the three overarching domains — program context, intervention, and implementation process — shown in the figure's gray boxes. Ultimately, as the rightmost oval shows, researchers were interested in measuring how participation in the SSPA programs may have influenced child and family outcomes.
In conducting the process evaluation, researchers examined the inputs into SSPA programs, such as the planning and start-up processes, organizational and program characteristics, staff, program costs, and collaboration with community partners. They also examined each site's activities, including the specific services provided, the implementation process, adjustments made during implementation, the barriers to and facilitators of program implementation, and any unexpected developments. Data collection for this process evaluation included site visits; quarterly site reports on service delivery, training, policies, and advocacy; a document review; regular email and telephone communication; and evaluation of staff training activities.
The evaluation identified many issues and yielded site-specific recommendations; here, we focus on the overall, transferable issues and recommendations related to the framework presented in the figure.
Recommendations About Program Context. The program context domain includes community setting, program setting, and evaluation design, and it encompasses factors that influence the implementation of community-based violence prevention programs, such as community support and readiness for a program and the fit between the program and the community. Table 1 presents some of the key cross-site issues and recommendations.
|Table 1. Key Program Context Recommendations|
Recommendations About Intervention. The intervention domain includes the intervention setting and approach, therapeutic content, case management or coordination, and other intervention components. Table 2 presents some of the key cross-site issues and recommendations.
|Table 2. Key Intervention Recommendations|
Recommendations About the Implementation Process. The implementation process domain includes referral and recruitment into the program, service delivery, family and child characteristics, and provider characteristics. Provider-level factors include staff selection, facilitative administration, and financial, organizational, and human resources systems. Table 3 presents some of the key cross-site issues and recommendations.
|Table 3. Key Implementation Process Recommendations|
As shown in the figure, the ultimate goal of the national evaluation was to assess whether programs at the 15 sites led to changes in child and family outcomes of interest. Those included eight outcome domains — (1) background and contextual factors (e.g., child and caregiver demographics), (2) PTSD, (3) depressive symptoms, (4) behavioral/conduct problems, (5) social-emotional competence, (6) caregiver-child relationship, (7) school readiness/performance, and (8) violence exposure — that were measured through a battery of measures relevant to the age of the child at baseline and at six, 12, 18, and 24 months after enrollment. The evaluators used an "intent-to-treat" approach designed to inform policymakers about the types of outcomes that can be expected if a similar program is implemented in a similar setting.
Overall, there were 18 separate evaluations of interventions within the 15 sites (three sites tested more than one intervention model), although one program stopped the evaluation early and is therefore not included in the outcomes report. Working with the national evaluation team, the sites completed a "green-light process" that developed the specific plans for the intervention and ensured that the evaluation plan would both align well with the intervention being offered and be ethical and feasible to implement. This process was completed before the programs were implemented to make sure that the evaluation and the program would be ready to launch simultaneously. As a result, a rigorous controlled evaluation design was developed at each site, either with a randomized control group (either a waitlist control group or an alternative intervention control group) — the gold standard of evaluation — or a comparison group that shared similar characteristics.
As is clear from the process evaluation results, many of programs at the sites had low enrollment rates, and retention and attrition in those programs were issues. Also, regardless of its program content or cost structure, each site received a standard funding allocation for providing services in its geographic area. It was a modest sum compared with that available in other types of clinical trials involving therapeutic interventions, especially in challenging real-world settings. Funding further constrained the evaluation when the federal appropriation for Safe Start was curtailed prematurely. Thus, follow-up assessments had to be cut short, sometimes even before one such assessment could be completed. This limited our ability to test whether effects might have been observed across a longer time frame.
Overall, because of underenrollment and attrition issues, modest evaluation funding, and the premature end of the evaluation, the retained sample in programs at most sites was smaller than the sample required to achieve the statistical power needed to detect the minimum observable effect. Specifically, sites conducting randomized trials enrolled between 16 and 91 percent of their targeted number of families, and those using comparison groups enrolled between 2 and 27 per- cent of their targets. In terms of retention, sites retained between 36 and 93 percent of the families originally enrolled (two sites that ended data collection early are not included in this result). Thus, all the studies were underpowered for the evaluation. When it comes to our ability to draw firm conclusions about intervention effects, we organize them into three categories of sites: those that are marginally powered, those that are underpowered, and those without a viable control or comparison condition.
Category 1: Sites with Programs Marginally Powered to Detect Intervention Effects in the Expected Range. For these sites and programs, the data can give some indication of the promise of the programs, but the intervention effects would need to be somewhat larger than originally expected to have a high probability (i.e., of greater than 80 percent) of being detected. Five sites fit into this category: the Bronx, Broward County, Dallas, Erie, and Kalamazoo.
In these sites, we observed significant improvements in outcomes between baseline and six months for some outcomes in the intervention group but not in the control group. These outcomes included caregiver reports of child PTSD symptoms (Erie, Kalamazoo), child behavioral/conduct problems (the Bronx), aspects of parenting stress (the Bronx, Erie), aspects of social-emotional competence (the Bronx, Broward County, Erie, Kalamazoo), caregivers' personal problems (Dallas), and aspects of academic achievement (Kalamazoo). However, in only one case did this improvement differ significantly from changes in the control or comparison groups: In Erie, there was a significant improvement attributable to participating in the intervention in one of the 18 primary outcomes identified, but it did not remain significant after controlling for demographic characteristics.
For other outcomes, improvement was observed in both the intervention and control groups, particularly in terms of reductions in types of violence exposure for both the child and the caregiver. However, these changes were partly the result of the time frame used in the scales that assessed violence exposure, with lifetime violence exposure being assessed at baseline and violence since baseline being assessed only at six months.
How much of the intended intervention services were delivered to families is also important. Because interventions were delivered in real-world community settings, families assigned to receive intervention services received them in varying amounts and combinations. Thus, the intent-to-treat analysis we conducted &mash; which answers the question of what outcomes to expect from a program implemented in a similar way — is conservative. Specifically, the intervention groups included some individuals who received no services or very few services. In many of the sites, only a minority of families received the full intervention as designed. To examine this issue, we used several different methods to compare those in the intervention group who received differing amounts of services with matched families from the comparison group who received none. We did not observe any obvious patterns that suggest that receiving more intervention services was related to a larger intervention effect. However, we did not have sufficient statistical power to be certain such a relationship does not exist.
Category 2: Sites with Programs Underpowered to Expect to Observe the Size of the Intervention Effects the Program Is Likely to Produce. For these sites and programs, we cannot know for sure whether the intervention was effective, because we did not have enough statistical power to detect the expected effects. The sites in this category were Dayton, Miami Infant Mental Health (IMH), Oakland, Providence Tier 3, San Diego, and San Mateo.
In these sites, we observed significant improvements in outcomes between baseline and six or 12 months for some outcomes in the intervention group but not in the control group. These included caregiver reports of child PTSD symptoms (San Diego), child behavioral/conduct problems (San Diego), aspects of social-emotional competence (Oakland), and caregivers' personal problems (Miami IMH). We also observed improvements on the child and caregiver violence exposure measures in both the intervention and control or comparison groups. As expected, we did not observe intervention effects for these programs.
Category 3: Sites Without a Viable Control or Comparison Condition. For such sites, inferences about the effect of the program cannot be made. These were Chelsea, Miami Heroes, Multnomah County, Providence Tier 2, and Washington Heights/Inwood. Although the Chelsea program showed significant improvements on several measures, we draw no conclusions about the intervention effects relative to a control or comparison condition. We also observed reductions in violence exposure in these sites, but these were expected, given the changes observed in the other sites being studied in both the intervention and control or comparison groups.
Overall, the evaluation produced a large number of inconclusive findings about the impact of interventions on child-level outcomes. In the few sites where meaningful statistical testing was possible, the lack of difference may mean that the interventions as implemented were simply not effective. However, several other explanations are equally possible. First, programs may have improved the lives of children and families in ways that were not measured (or that were measured inadequately); because measures were uniform across the evaluation rather than tailored to each program, some measures may not have been finely tuned enough to detect intervention effects at certain sites.
A second possibility has to do with the therapeutic process and the timing of assessments. Several sites noted that families were sometimes unaware of the connection between violence exposure and a child's symptoms when they entered the program, and they became aware of this link only as they learned more during the intervention process. If this is true, subsequent reports of child behavior and symptoms could, in comparison with the baseline, have been heightened because the caregiver was more aware of and sensitive to them. This would obscure any intervention effects. Similarly, some interventions could, as part of the therapeutic process, have exacerbated symptoms before reducing them. If so, only the later assessments (i.e., at 12, 18, or 24 months) would have shown the full impact of the program, and the six-month assessments could be misleading.
Third, the way the programs were implemented could have affected the findings. For example, in some sites, the control group received enhanced services, which may have reduced the amount of difference between the two groups, making an intervention effect more difficult to detect. Moreover, differential retention in the two groups at some sites may mean that selection bias played a role in the outcomes observed. For example, families who were having difficulty may have been more difficult to reach and assess, thereby biasing outcomes in those assessed. Another possibility is that intervention group families were unable to engage in the services because of extremely challenging life circumstances or because of a mismatch between their perceived needs and the program services. Finally, participants in a few sites had low levels of symptoms at baseline, making it difficult to demonstrate changes in children over time: Children who were not experiencing many symptoms or problems at baseline may not have had "room to improve." In such instances, services could potentially provide a longer-term protective benefit that could not be observed in this evaluation.
As the focus on CEV has increased among researchers and public agencies — and as its negative consequences on health, behavior, and development have become more evident — intervention programs have been developed to try to improve outcomes for these children. However, many of these programs lack evidence of efficacy or effectiveness on child-level outcomes, and the few that have been empirically evaluated have not been well tested in real-world community settings. The SSPA evaluation is therefore important because it attempted to rigorously examine the effectiveness of such programs delivered in community settings, even with relatively modest funding levels. The level of rigor in the evaluation we attempted is rarely seen in community implementation projects.
The diversity of the SSPA programs under study is also noteworthy. Although all of the programs focused on CEV, they varied considerably in terms of intervention type, the setting in which they were offered, and the group of children and families they targeted. All the programs were able to successfully launch and provide some services to children and families exposed to violence.
The programs presented many obstacles to successful evaluation, and these were at least partially surmounted. Examples of successes included developing measures that could assess the sensitive topic of violence exposure and examine outcomes across a range of ages, using advanced psychometric techniques to develop measures that would span a broader age range, working with multiple Institutional Review Boards to ensure the confidentiality of the data collected and define the limits of confidentiality, and engaging community partners and families in the evaluation studies.
Other obstacles were harder to overcome, including funding limits, low uptake of the intervention services in some cases, and struggles with recruitment and retention. All the challenges weakened our ability to draw firm conclusions from the data collected. However, despite the limitations, these data will be useful in planning future research endeavors and learning more about CEV and affected families. The difficulties we faced in conducting this outcome evaluation will also provide useful information about the types of challenges faced by families entering intervention services and the challenges involved in evaluating child outcomes in a variety of settings.
In summary, although the outcomes evaluation overall produced no conclusive findings about the effectiveness of the sites' interventions, the SSPA process evaluation showed that individual programs experienced many successes in implementing their goals. These successes included developing procedures for increased identification of CEV, improving communication and coordination among service providers, and establishing new interagency and communitywide partnerships to address service gaps for children and their families. These advances have value in their own right and should be considered part of the legacy of the Safe Start Initiative.
 Statistics are from Department of Health and Human Services, Administration on Children, Youth and Families, Child Maltreatment 2008, Washington, D.C.: U.S. Government Printing Office, 2010; and K. Baum, Juvenile Victimization and Offending, 1993–2003, Bureau of Justice Statistics Special Report, NCJ 209468, 2005.