Aug 25, 2011
Results of the National Evaluation of Safe Start Promising Approaches
Published Sep 13, 2011
RAND conducted the national evaluation of Safe Start Promising Approaches (SSPA)—an initiative that involved 15 sites implementing promising and evidence-based children's exposure to violence (CEV) programs in community settings—to identify how well such programs work in reducing and preventing CEV's harmful effects. Overall, because of both methodological and programmatic issues, the outcomes evaluation revealed many inconclusive findings about the impact of interventions on child-level outcomes. However, the SSPA process evaluation identified many successes among the individual programs in implementing program goals, including developing procedures for increased CEV identification, improving communication and coordination among service providers, and establishing new interagency and communitywide partnerships to address service gaps for children and their families. These improvements have value in their own right and should be considered part of the legacy of the Safe Start Initiative.
In recent years, approximately 61 percent of children in the United States reported that they were exposed to violence, crime, and abuse during the previous year. Common sources of children's exposure to violence (CEV) are direct child maltreatment, witnessing domestic violence, and community and school violence. There is reason to believe the problem may have grown worse in recent decades. Child protective services agencies accepted 3.3 million referrals for neglect and abuse in 2008, and, from these referrals, 772,000 children were found to be victims of abuse or neglect. This figure represents a national rate of 10.3 victims per 1,000 children. Also, a high proportion of children are exposed to violence that occurs between adult intimate partners. Finally, exposure to violence outside the home is also common, with youth being victims of violent crime at a rate of 26.5 per 1,000 in 2009.
CEV also represents a risk factor for a variety of problems, including psychiatric disorders and behavioral problems, such as post-traumatic stress disorder (PTSD), depression, anxiety, and developmental problems. School performance has also been shown to suffer as a result of CEV. Research suggests that the effects of exposure to violence may persist well into adulthood.
Given the risk to children's health and well-being from CEV, there is a real need for effective prevention and intervention programs. But, despite some progress in identifying evidence-based models and practices, programs to date are limited in their reach (e.g., are limited to particular age ranges, settings, or symptoms), and few have been well tested in community settings. Demonstration projects have rarely moved beyond the description phase and into rigorous test interventions and impact evaluations. This means that community practitioners have little research to draw on when attempting to address the needs of the trauma-exposed families with whom they work.
The need both to develop better CEV programs and practices and to demonstrate that they can work in community settings was the impetus for the Safe Start Initiative launched by the Office of Juvenile Justice and Delinquency Prevention (OJJDP) in 2000 as a community-based initiative focused on developing, fielding, and evaluating interventions to prevent and reduce the impact of CEV. The initiative consists of four phases, including a first phase (completed in 2006) involving demonstrations of various innovative, CEV-related promising practices in the system of care.
The RAND Corporation was selected to conduct the national evaluation of the second phase—the implementation of 15 Safe Start Promising Approaches (SSPA) CEV programs in community settings—to identify how well such programs work in reducing and preventing CEV's harmful effects. This research highlight summarizes the results of the process and outcomes evaluations published in two RAND reports.
In 2005, OJJDP selected 15 SSPA program sites spanning cities and counties across the country: the Bronx, New York; Broward County, Florida; Chelsea, Massachusetts; Dallas, Texas; Dayton, Ohio; Erie, Pennsylvania; Kalamazoo, Michigan; Miami, Florida; Multnomah County, Oregon; Oakland, California; Providence, Rhode Island; San Diego County, California; San Mateo County, California; Toledo, Ohio; and Washington Heights/Inwood, New York. The sites drew from the demonstrations of innovative promising approaches in the first phase and were selected based on the strength of the evidence and theory base of their designs and on the feasibility of the interventions.
The 15 sites were also diverse in terms of community size, populations served, types of violence exposure, intervention strategies, and implementation settings. For example, community sizes ranged from those with catchment areas with fewer than 500,000 residents to those with more than 1 million residents. The sites were also racially and ethnically diverse, ranging from 85 percent white in Kalamazoo to 36 percent black in Oakland to 57 percent Hispanic in Miami.
Within their communities, the SSPA programs were situated in or overseen by a variety of different lead agencies or organizations, ranging from health clinics or hospitals to domestic violence agencies to human services agencies. SSPA sites varied in terms of the age range of children served: Most served families with young children, but some also served families with children up to 18 years old. Targeted populations were also diverse, ranging from children exposed to domestic violence or other forms of violence to children exiting domestic violence shelters to children who were victims of abuse and neglect. The referral agencies themselves were also diverse, ranging from clinics and hospitals to domestic violence shelters to Head Start classrooms.
To understand and learn from the variety of interventions in community settings, researchers grounded their evaluations in a conceptual model (shown in the figure) that was adapted from other research that involved evaluating the implementation of community-based mental health programs. In this model, a planned program includes such components as program design and target population. Planned implementation support includes preplanning activities, the quality of program materials, technical support, and implementer readiness. Both the planned program and the planned implementation support feed into the program as it is actually implemented.
The program as it is implemented is also influenced by other factors: community, provider, intervention, delivery system, and support system characteristics. The researchers organized these factors into the three overarching domains—program context, intervention, and implementation process—shown in the figure's gray boxes. Ultimately, as the rightmost oval shows, researchers were interested in measuring how participation in the SSPA programs may have influenced child and family outcomes.
In conducting the process evaluation, researchers examined the inputs into SSPA programs, such as the planning and start-up processes, organizational and program characteristics, staff, program costs, and collaboration with community partners. They also examined each site's activities, including the specific services provided, the implementation process, adjustments made during implementation, the barriers to and facilitators of program implementation, and any unexpected developments. Data collection for this process evaluation included site visits; quarterly site reports on service delivery, training, policies, and advocacy; a document review; regular email and telephone communication; and evaluation of staff training activities.
The evaluation identified many issues and yielded site-specific recommendations; here, we focus on the overall, transferable issues and recommendations related to the framework presented in the figure.
Recommendations About Program Context. The program context domain includes community setting, program setting, and evaluation design, and it encompasses factors that influence the implementation of community-based violence prevention programs, such as community support and readiness for a program and the fit between the program and the community. Table 1 presents some of the key cross-site issues and recommendations.
|Importance of widespread recognition of need
|Provide opportunities to bring community agencies and service providers together to identify and address gaps in and barriers to services for children exposed to violence. Collaboration is a promising way to develop a program that addresses gaps in the existing network of services and supports and fits the needs of children exposed to violence.
|Importance of strong individual leadership at the lead agency
|Dedicate a staff leader with the power to make program-level decisions and adjustments. A program leader with adequate time to lead can effectively develop collaborative relationships with community partners and implement the program. Ideally, the program leader would have decisionmaking authority for program activities and modifications.
|Importance of a clear division of responsibility for program implementation
|Design a program that places primary responsibility for implementation with a lead agency or organization. The program should be structured with a clearly delineated lead agency or organization.
|Importance of close working relationships among partner agencies and referral sources
|Capitalize on existing strong relationships among partner agencies and referral sources. Those involved in the program's design and planning phases should take advantage of existing collaborative efforts and relationships and prior experiences together. Trust between agencies appears to be particularly important for interventions for children exposed to violence; it can increase initial buy-in and ensure that partner agencies understand why supports and services are needed and how the planned program is expected to affect outcomes for participating families and children.
|Select known or internal referral sources. To the extent possible, take advantage of existing relationships among community agencies or organizations or internal referral sources. This can help facilitate program implementation.
|Burden of research requirements
|Assess the impact of the evaluation design on program staff, families, and referral sources. Given an evaluation component, first recognize and understand how the specific research plan affects different parties, including the program staff providing supports and services, the participating families, and the staff at the referral sources.
|Develop tools to increase buy-in for the evaluation component and to minimize negative impact. Next, educate all parties about the benefits of the research project, including its role in augmenting available community services and ensuring effective treatment. Education tools might include program brochures or fact sheets geared toward the intended audience that use nontechnical language to explain the research design and the importance of the research component to the overall program.
Recommendations About Intervention. The intervention domain includes the intervention setting and approach, therapeutic content, case management or coordination, and other intervention components. Table 2 presents some of the key cross-site issues and recommendations.
|Problem with differences between the intervention approaches and the families' needs and priorities
|Understand the priorities and needs of families in the program's target population. When selecting the intervention components, align the overall intervention approach and the mix of services and supports offered with the needs and issues of the families with a child exposed to violence. Despite the identified violence exposure, families may have more-pressing basic needs that the program needs to tackle before addressing the violence exposure.
|Importance of a convenient intervention setting
|Consider the intervention setting. Service delivery location has implications for implementation. To help engage families and address some of the safety, transportation, and child care issues related to participating in a mental health program, it may be beneficial to offer the program in the home or in a school or daycare setting. Particularly for families with small children, a convenient intervention setting can help ease the burden of participating in mental health treatment and can increase the families' engagement.
|Importance of needs assessments or developmental screenings
|Conduct needs assessments or developmental screening to engage families. An intervention approach that thoroughly assesses the families' needs can help ensure that the overall intervention approach aligns with families' circumstances, including their safety issues related to violence.
|Importance of providing for families' immediate and basic needs
|Ensure that families are provided for or connected to sources of assistance to meet basic needs. Many families with children exposed to violence have multiple stressors, potentially making it difficult to focus on and engage in mental health treatment. At program outset, assess families' basic needs for services and supports and either provide them with case management or coordination or refer them to an agency or organization that can help address some immediate needs.
|Importance of an array of therapy options
|Offer more than one therapeutic approach to facilitate matching services to individual families' needs and circumstances. No one therapeutic approach is appropriate for all families with children exposed to violence. By assessing families' therapeutic needs and offering multiple therapeutic options that can meet those needs, the program can better address different families' unique needs.
|Importance of adaptable components of the therapy models
|Modify the selected therapeutic model to offer activities or a curriculum most suitable for the target population. Certain therapeutic models can be used flexibly. If the therapeutic model allows, tailor certain activities, examples, or techniques to help the program better meet families' needs. Consult with experts before making modifications to ensure that the core model components are retained.
|Problem with monitoring adherence to the therapy models
|Develop a process for monitoring adherence to the therapy model over time. To maximize the effectiveness of the therapeutic model, integrate methods for monitoring how program staff providing treatment are following intervention guidelines or standards. By designating someone to take responsibility for adherence, the program administrators can assess whether the program staff delivering the intervention are adequately trained, demonstrate acceptable skill levels in delivering the intervention services, and adhere to intervention standards described in the site's model.
Recommendations About the Implementation Process. The implementation process domain includes referral and recruitment into the program, service delivery, family and child characteristics, and provider characteristics. Provider-level factors include staff selection, facilitative administration, and financial, organizational, and human resources systems. Table 3 presents some of the key cross-site issues and recommendations.
|Importance of close physical proximity of referral sources and program staff
|Develop internal agency referral sources or co-locate program staff with the referral source. Close proximity of the referring party to program staff can help ensure a steady referral stream. In the case of programs that receive some or all referrals internally, staff can be educated about the program and the mechanics of the referral process. In the case of programs that receive external referrals, consider placing program staff on site with the referral agency or organization to facilitate referrals.
|Importance of incentivizing referral sources
|Provide training and outreach to referral sources. Agencies and organizations providing referrals or conducting program recruitment must understand how the planned program is expected to affect outcomes related to violence exposure for participating families and children and how the program benefits their agency or organization. Education and outreach efforts can include program materials, training sessions, and one-on-one meetings with staff making referrals.
|Importance of stable referral sources
|Prepare for staff turnover at referral agencies. Develop program materials and referral processes that can be transferred as referral agency staff turn over. Program materials, referral forms, contact information, and communication pathways must be clear, concise, and readily accessible and transferable for existing and new staff at the referral agency or organization.
|Importance of ongoing training of staff members
|Plan for training over time rather than simply at the beginning. Our training evaluation revealed that early gains in knowledge in serving children exposed to violence are not sustained over time; thus, booster sessions to maintain skills and comfort levels may be needed, particularly in the area of family engagement.
|Problem of cumbersome referral processes
|Streamline the referral process. Make the referral process as simple and straightforward as possible. This will allow the referring agencies to bring interested families into the intervention as quickly as possible after referral for (or detection of) violence exposure.
|Problem of highly mobile families
|Recruit families from multiple systems of care and use multiple approaches to track and locate families. High rates of family transience, partially related to violence exposure, can reduce program recruitment and retention. Recruitment from more than one system of care may broaden outreach to eligible families and increase the number of families served. Also, collect abundant information on families at intake so as to be able to "track and locate" them over time.
|Problem of families' negative views of or experiences with mental health services referrals
|Work with referral agencies to address issues related to the stigma of mental health services. To reduce the stigma associated with mental health services and to increase families' interest in accepting mental health services referrals, those who are making referrals or recruiting families to participate need to frame the program as something that will help the families' children.
|Problem of families' complex treatment needs
|Develop a complete treatment plan to address all family needs. Program staff and service providers should develop a comprehensive plan for services and supports with clearly defined responsibilities and lines of communication to coordinate among the different social service organizations involved. Also, provide up-front case management and support to families to address immediate and basic family needs related to exposure to violence and other disadvantages.
|Importance of close relationships with families
|Work with referral sources and service providers on strategies to build relationships, establish trust, and help families understand the potential benefits of mental health services. Those making referrals must be educated on recognizing CEV and symptoms of trauma so that they feel comfortable discussing these issues. The referral process should allow enough time for those making referrals to establish a trusting relationship with families. To help develop strong connections between families and the program's service providers, consider including an assessment and relationship-building component that enables families to become familiar with and accepting of providers.
|Problem of culturally inconsistent services
|Educate program staff about cultural differences and develop service delivery approaches that respect cultural issues. To address cultural aspects of service delivery for a target population, the lead agency or organization should employ staff and partner with agencies with different cultural and language competencies.
|Problem of unclear program parameters
|Define the roles of different service providers. The program can be structured so that those providing services have clearly defined roles and so that the criteria for ending services are clear. Families must understand who is responsible for different intervention components and how progress will be assessed.
|Importance of a flexible service delivery model
|Adjust service delivery processes to help program staff manage time and caseloads. A flexible service delivery model can enable a program's service providers to change the setting, staff hours, staff roles, or caseloads to serve more families and better manage time and caseloads.
|Problem of staffing issues among service providers
|Maintain regular contact with program staff and provide routine refresher training and supervision. The program's clinical supervisors should also monitor caseloads, staff morale and burnout, and the quality of the services being offered.
|Importance of close communication and coordination between program staff and service providers
|Plan multidisciplinary or case coordination meetings. To facilitate understanding and communication and provide a forum for troubleshooting, program staff and service providers can plan regular case management or coordination meetings that allow those involved to share information, discuss the family's situation, ask questions of one another, and plan next steps.
|Problem of strained relationships among partner agencies
|Recognize the different perspectives of partner agencies or organizations. In planning the program, recognize that some partner agencies have different perspectives of and orientations toward CEV.
As shown in the figure, the ultimate goal of the national evaluation was to assess whether programs at the 15 sites led to changes in child and family outcomes of interest. Those included eight outcome domains—(1) background and contextual factors (e.g., child and caregiver demographics), (2) PTSD, (3) depressive symptoms, (4) behavioral/conduct problems, (5) social-emotional competence, (6) caregiver-child relationship, (7) school readiness/performance, and (8) violence exposure—that were measured through a battery of measures relevant to the age of the child at baseline and at six, 12, 18, and 24 months after enrollment. The evaluators used an "intent-to-treat" approach designed to inform policymakers about the types of outcomes that can be expected if a similar program is implemented in a similar setting.
Overall, there were 18 separate evaluations of interventions within the 15 sites (three sites tested more than one intervention model), although one program stopped the evaluation early and is therefore not included in the outcomes report. Working with the national evaluation team, the sites completed a "green-light process" that developed the specific plans for the intervention and ensured that the evaluation plan would both align well with the intervention being offered and be ethical and feasible to implement. This process was completed before the programs were implemented to make sure that the evaluation and the program would be ready to launch simultaneously. As a result, a rigorous controlled evaluation design was developed at each site, either with a randomized control group (either a waitlist control group or an alternative intervention control group)—the gold standard of evaluation—or a comparison group that shared similar characteristics.
As is clear from the process evaluation results, many of programs at the sites had low enrollment rates, and retention and attrition in those programs were issues. Also, regardless of its program content or cost structure, each site received a standard funding allocation for providing services in its geographic area. It was a modest sum compared with that available in other types of clinical trials involving therapeutic interventions, especially in challenging real-world settings. Funding further constrained the evaluation when the federal appropriation for Safe Start was curtailed prematurely. Thus, follow-up assessments had to be cut short, sometimes even before one such assessment could be completed. This limited our ability to test whether effects might have been observed across a longer time frame.
Overall, because of underenrollment and attrition issues, modest evaluation funding, and the premature end of the evaluation, the retained sample in programs at most sites was smaller than the sample required to achieve the statistical power needed to detect the minimum observable effect. Specifically, sites conducting randomized trials enrolled between 16 and 91 percent of their targeted number of families, and those using comparison groups enrolled between 2 and 27 percent of their targets. In terms of retention, sites retained between 36 and 93 percent of the families originally enrolled (two sites that ended data collection early are not included in this result). Thus, all the studies were underpowered for the evaluation. When it comes to our ability to draw firm conclusions about intervention effects, we organize them into three categories of sites: those that are marginally powered, those that are underpowered, and those without a viable control or comparison condition.
Category 1: Sites with Programs Marginally Powered to Detect Intervention Effects in the Expected Range. For these sites and programs, the data can give some indication of the promise of the programs, but the intervention effects would need to be somewhat larger than originally expected to have a high probability (i.e., of greater than 80 percent) of being detected. Five sites fit into this category: the Bronx, Broward County, Dallas, Erie, and Kalamazoo.
In these sites, we observed significant improvements in outcomes between baseline and six months for some outcomes in the intervention group but not in the control group. These outcomes included caregiver reports of child PTSD symptoms (Erie, Kalamazoo), child behavioral/conduct problems (the Bronx), aspects of parenting stress (the Bronx, Erie), aspects of social-emotional competence (the Bronx, Broward County, Erie, Kalamazoo), caregivers' personal problems (Dallas), and aspects of academic achievement (Kalamazoo). However, in only one case did this improvement differ significantly from changes in the control or comparison groups: In Erie, there was a significant improvement attributable to participating in the intervention in one of the 18 primary outcomes identified, but it did not remain significant after controlling for demographic characteristics.
For other outcomes, improvement was observed in both the intervention and control groups, particularly in terms of reductions in types of violence exposure for both the child and the caregiver. However, these changes were partly the result of the time frame used in the scales that assessed violence exposure, with lifetime violence exposure being assessed at baseline and violence since baseline being assessed only at six months.
How much of the intended intervention services were delivered to families is also important. Because interventions were delivered in real-world community settings, families assigned to receive intervention services received them in varying amounts and combinations. Thus, the intent-to-treat analysis we conducted—which answers the question of what outcomes to expect from a program implemented in a similar way—is conservative. Specifically, the intervention groups included some individuals who received no services or very few services. In many of the sites, only a minority of families received the full intervention as designed. To examine this issue, we used several different methods to compare those in the intervention group who received differing amounts of services with matched families from the comparison group who received none. We did not observe any obvious patterns that suggest that receiving more intervention services was related to a larger intervention effect. However, we did not have sufficient statistical power to be certain such a relationship does not exist.
Category 2: Sites with Programs Underpowered to Expect to Observe the Size of the Intervention Effects the Program Is Likely to Produce. For these sites and programs, we cannot know for sure whether the intervention was effective, because we did not have enough statistical power to detect the expected effects. The sites in this category were Dayton, Miami Infant Mental Health (IMH), Oakland, Providence Tier 3, San Diego, and San Mateo.
In these sites, we observed significant improvements in outcomes between baseline and six or 12 months for some outcomes in the intervention group but not in the control group. These included caregiver reports of child PTSD symptoms (San Diego), child behavioral/conduct problems (San Diego), aspects of social-emotional competence (Oakland), and caregivers' personal problems (Miami IMH). We also observed improvements on the child and caregiver violence exposure measures in both the intervention and control or comparison groups. As expected, we did not observe intervention effects for these programs.
Category 3: Sites Without a Viable Control or Comparison Condition. For such sites, inferences about the effect of the program cannot be made. These were Chelsea, Miami Heroes, Multnomah County, Providence Tier 2, and Washington Heights/Inwood. Although the Chelsea program showed significant improvements on several measures, we draw no conclusions about the intervention effects relative to a control or comparison condition. We also observed reductions in violence exposure in these sites, but these were expected, given the changes observed in the other sites being studied in both the intervention and control or comparison groups.
Overall, the evaluation produced a large number of inconclusive findings about the impact of interventions on child-level outcomes. In the few sites where meaningful statistical testing was possible, the lack of difference may mean that the interventions as implemented were simply not effective. However, several other explanations are equally possible. First, programs may have improved the lives of children and families in ways that were not measured (or that were measured inadequately); because measures were uniform across the evaluation rather than tailored to each program, some measures may not have been finely tuned enough to detect intervention effects at certain sites.
A second possibility has to do with the therapeutic process and the timing of assessments. Several sites noted that families were sometimes unaware of the connection between violence exposure and a child's symptoms when they entered the program, and they became aware of this link only as they learned more during the intervention process. If this is true, subsequent reports of child behavior and symptoms could, in comparison with the baseline, have been heightened because the caregiver was more aware of and sensitive to them. This would obscure any intervention effects. Similarly, some interventions could, as part of the therapeutic process, have exacerbated symptoms before reducing them. If so, only the later assessments (i.e., at 12, 18, or 24 months) would have shown the full impact of the program, and the six-month assessments could be misleading.
Third, the way the programs were implemented could have affected the findings. For example, in some sites, the control group received enhanced services, which may have reduced the amount of difference between the two groups, making an intervention effect more difficult to detect. Moreover, differential retention in the two groups at some sites may mean that selection bias played a role in the outcomes observed. For example, families who were having difficulty may have been more difficult to reach and assess, thereby biasing outcomes in those assessed. Another possibility is that intervention group families were unable to engage in the services because of extremely challenging life circumstances or because of a mismatch between their perceived needs and the program services. Finally, participants in a few sites had low levels of symptoms at baseline, making it difficult to demonstrate changes in children over time: Children who were not experiencing many symptoms or problems at baseline may not have had "room to improve." In such instances, services could potentially provide a longer-term protective benefit that could not be observed in this evaluation.
As the focus on CEV has increased among researchers and public agencies—and as its negative consequences on health, behavior, and development have become more evident—intervention programs have been developed to try to improve outcomes for these children. However, many of these programs lack evidence of efficacy or effectiveness on child-level outcomes, and the few that have been empirically evaluated have not been well tested in real-world community settings. The SSPA evaluation is therefore important because it attempted to rigorously examine the effectiveness of such programs delivered in community settings, even with relatively modest funding levels. The level of rigor in the evaluation we attempted is rarely seen in community implementation projects.
The diversity of the SSPA programs under study is also noteworthy. Although all of the programs focused on CEV, they varied considerably in terms of intervention type, the setting in which they were offered, and the group of children and families they targeted. All the programs were able to successfully launch and provide some services to children and families exposed to violence.
The programs presented many obstacles to successful evaluation, and these were at least partially surmounted. Examples of successes included developing measures that could assess the sensitive topic of violence exposure and examine outcomes across a range of ages, using advanced psychometric techniques to develop measures that would span a broader age range, working with multiple Institutional Review Boards to ensure the confidentiality of the data collected and define the limits of confidentiality, and engaging community partners and families in the evaluation studies.
Other obstacles were harder to overcome, including funding limits, low uptake of the intervention services in some cases, and struggles with recruitment and retention. All the challenges weakened our ability to draw firm conclusions from the data collected. However, despite the limitations, these data will be useful in planning future research endeavors and learning more about CEV and affected families. The difficulties we faced in conducting this outcome evaluation will also provide useful information about the types of challenges faced by families entering intervention services and the challenges involved in evaluating child outcomes in a variety of settings.
In summary, although the outcomes evaluation overall produced no conclusive findings about the effectiveness of the sites' interventions, the SSPA process evaluation showed that individual programs experienced many successes in implementing their goals. These successes included developing procedures for increased identification of CEV, improving communication and coordination among service providers, and establishing new interagency and communitywide partnerships to address service gaps for children and their families. These advances have value in their own right and should be considered part of the legacy of the Safe Start Initiative.