National Evaluation of Safe Start Promising Approaches

Nationally, approximately 61 percent of children have been exposed to violence during the past year, and there is reason to believe that the problem has grown worse in recent decades (Finkelhor et al., 2009). Children's exposure to violence (CEV) can have serious consequences, including a variety of psychiatric disorders and behavioral problems, such as posttraumatic stress disorder (PTSD), depression, and anxiety. School performance has also been shown to suffer as a result of CEV. Moreover, research suggests that the effects of exposure to violence may persist well into adulthood. Fortunately, research has also shown that early childhood interventions can substantially improve children's chances of future social and psychological well-being.

Background: The Safe Start Program

Safe Start is a community-based initiative focused on developing and fielding interventions to prevent and reduce the impact of CEV. Sponsored by the U.S. Department of Justice's Office of Juvenile Justice and Delinquency Prevention (OJJDP), the initiative consists of four phases:

• Phase One: Expanding the system of care for children exposed to violence by conducting demonstration projects. Now complete, this phase involved demonstrations of various innovative promising practices in the system of care for children who have been exposed to violence.
• Phase Two: Building on Phase One, this phase was intended to implement and evaluate promising and evidence-based programs in community settings to identify how well programs worked in reducing and preventing the harmful effects of CEV.
• Phases Three: Still in the planning stages, the aim of this phase is to build a knowledge base of effective interventions in the field of CEV.
• Phase Four: The goal of this phase is to “seed” on a national scale the effective strategies identified in the earlier phases.

Each phase also includes an evaluation component, intended to assess the implementation of the various interventions and their impact on children's outcomes.

This article focuses on Phase Two of the Safe Start program. For this phase, known as Safe Start Promising Approaches (SSPA), OJJDP selected 15 program sites across the country to implement a range of interventions for helping children and families cope with the effects of CEV. Sites were located in the following communities:

• Bronx, New York
• Broward County, Florida
• Chelsea, Massachusetts
• Dallas, Texas
• Dayton, Ohio
• Erie, Pennsylvania
• Kalamazoo, Michigan
• Miami, Florida
• Multnomah County, Oregon
• Oakland, California
• Providence, Rhode Island
• San Diego County, California
• San Mateo County, California
• Toledo, Ohio
• Washington Heights/Inwood, New York.

The program sites varied in numerous ways. First, they focused on multiple types of violence exposure and interventions for such exposure, including variations in ages and age-appropriate practices. The 15 sites also varied in size, location, and population characteristics. Each of the communities had identified barriers to services for children exposed to violence and viewed SSPA as an opportunity to increase capacity, coordinate services, and address gaps in the array of services in the community. The SSPA programs were situated locally within a variety of different kinds of lead agencies or organizations, including health clinics or hospitals, human services agencies, organizational units within universities, domestic violence or child maltreatment services agencies, and county-level government offices. In developing their programs, the lead agencies partnered with the specific agencies in the community that work with children exposed to violence: law enforcement agencies, child protective services agencies, human services agencies, behavioral health organizations, and other community nonprofit agencies. The programs received their referrals from a variety of sources, including the clinic or hospital system, child welfare system, domestic violence shelters, human services agencies, Head Start, or a combination of the above.

The 15 Safe Start programs comprised a range of intervention components. All included a therapeutic component; about two-thirds focused on dyadic or family therapy, while the rest used individual or group therapy approaches. In some cases, the modality varied by age, with dyadic or family therapy for younger children and group therapy for older children. Many of the sites also used case management or coordination. Some of the sites had other intervention components, such as advocacy, parent groups, or other services (e.g., multidisciplinary evaluation of family needs, an in-home safety assessment, etc.). The intervention setting also varied, with interventions offered in families' homes, clinics, shelters, child centers, or Head Start classrooms. The interventions varied in length from three months to more than one year. At all of the sites, the interventions were conducted in the context of a rigorous evaluation, as required by OJJDP.

Evaluation Approach

RAND Corporation researchers evaluated the SSPA phase of the initiative in collaboration with the national evaluation team: OJJDP, the Safe Start Center, the Association for the Study and Development of Communities (ASDC), and the 15 program sites. The evaluation design involved three components: a process evaluation, including a cost analysis; an evaluation of training; and an outcomes evaluation. This article presents the results of our implementation process evaluation as well as the cost and training evaluation results.

Our evaluation of the SSPA programs drew on a framework that focused on three domains: the program context, the intervention, and the process of implementation followed by the programs.

The program context domain includes community setting, program setting, and evaluation design. This domain encompasses factors such as community support and readiness for a program and the fit between the program and the community that influence the implementation of community-based violence prevention programs. The intervention domain includes the intervention setting and approach, therapeutic content, case management or coordination, and other intervention components. Finally, the implementation process includes referral and recruitment into the program and service delivery in addition to the family/child characteristics and provider characteristics. Provider-level factors include staff selection, facilitative administration, and financial, organizational, and human resources systems.

To collect information about the sites and the interventions, we examined the inputs into SSPA projects, such as the planning and start-up processes, organizational and program characteristics, staff, costs, and collaboration with community partners. We also examined each site's activities, including the specific services provided, the implementation process, adjustments made during implementation, the barriers and facilitators to program implementation, and any unexpected developments.

Data collection for this process evaluation included site visits; quarterly activity reports on services, training, policies, and advocacy; document review; regular email and telephone communication; and evaluation of staff training activities. During the site visits, we collected data from a variety of sources, including key informant interviews with program staff and community partners, case reviews of randomly selected cases, tours of facilities, a quality assurance checklist completed by the clinical supervisor, and observations of relevant meetings.

Findings: Factors Affecting SSPA Implementation

Our assessment of the program context, interventions, and implementation processes identified factors that were related to implementation of the SSPA programs. The programs' focus on children exposed to violence led to the identification of some unique factors that are most relevant to other such programs, but we also identified factors that are generic to rolling out any mental health program in community settings. Some factors helped implementation, and some hindered it. We summarize these factors below, highlighting those that were evident across more than one site or that offered particularly clear implications for future work with children exposed to violence.

Program Context

The following contextual factors were found to affect program implementation at some sites:

• Widespread recognition of need. When there was widespread recognition among community agencies and service providers of the need to increase capacity and coordinate services, the lead agencies/organizations were better able to develop an intervention that addressed the gaps in the array of services for children exposed to violence.
• Strong individual leadership at the lead agency. Strong individual leaders at some SSPA programs served as program advocates in the community and within the lead agency/organization. These leaders were able to increase the program's visibility both externally and internally and provide support and direction for program implementation.
• Clear division of responsibility for program implementation. The structure of some of the lead agencies or organizations meant that they could take responsibility for providing the staff and resources for program implementation. In turn, partner agencies were not burdened with responsibilities related to implementing the programs.
• Close working relationships among partner agencies and referral sources. Some SSPA programs took advantage of existing relationships and trust developed through prior experiences with partner agencies or organizations to smooth the way for implementing a new program in the community. These types of relationships were evidenced by strong buy-in from partner organizations, demonstrated by up-front support and understanding of the potential program benefits.
• Burden of research requirements. SSPA's research component posed challenges for program implementation. The reluctance of referral sources and families to participate when the program involved a control or comparison group negatively influenced the referral and recruitment processes, even though the services being offered were not otherwise available in the community.

Based on these factors, we drew implications for program context that other communities seeking to implement similar interventions can use to facilitate implementation. Table 1 recaps the factors discussed above and pairs them with relevant implications.

Intervention

The following factors related to the interventions themselves were found to affect implementation across more than one site:

• Differences between the intervention approaches and the families' needs and priorities. While the intervention approaches were developed to fill a gap in mental health services in the community as perceived by community partners, the focus on mental health services did not necessarily fit with families' own priorities, which were often directed toward addressing safety and basic needs before turning to mental health. This issue with sequencing of services created challenges with service delivery, as families had other priorities at the time of their referral into the programs.
• Convenience of intervention setting. Many of the SSPA programs delivered the services in families' homes that enabled them to address some of the logistical challenges related to successful engagement of families in treatment and provide a safe environment for families to receive services.
• Use of needs assessment or developmental screenings. SSPA programs that included a needs assessment or development screening as one of their intervention components found that this approach helped them engage families.
• Ability to provide for families' immediate and basic needs. The case management or coordination component enabled the SSPA programs to connect with families and address their basic needs. Doing so helped providers engage with families who were unfamiliar with mental health and begin to establish trust and increase comfort with the intervention.
• Wide array of therapy options. By offering a wide range of therapy options, some SSPA programs were able to tailor the therapeutic component to the families' needs and circumstances. This sort of flexible approach helped these programs be responsive to the unique needs of families.
• Adaptable components of the therapy models. Some therapy models provided guidelines rather than manuals. This enabled the clinicians to tailor the therapeutic strategies and techniques to the families' circumstances.
• Inability to monitor adherence to the therapy models. Because of resource constraints, most SSPA programs were unable to systematically ascertain whether individual project staff were following the therapeutic guidelines or standards when delivering the intervention services. This made it difficult to assess whether the therapy was delivered as intended by the model developers.

The factors and their implications for developing and implementing community-based programs for children exposed to violence are summarized in Table 2.

Implementation Process

The following factors related to the process of implementing the interventions were commonly found to affect implementation at the sites:

• Close physical proximity of referral sources and program staff. Some SSPA programs were structured so that referral sources were located at the program office or program staff regularly went on-site to the referral agency to identify referrals. Referrals from internal or co-located staff at the referral source made it easier for program staff and referral sources to communicate and resulted in a steadier flow of referrals.
• Provision of incentives (e.g., training) to the referral sources. Some referral sources had been trained by program staff about the issues of CEV and the program's benefits and understood the potential positive impact of the program on families. These referral sources were better positioned to maintain a steady flow of referrals.
• Stable referral sources. Some SSPA referral agencies experienced frequent staff turnover, making it difficult to maintain a steady flow of referrals and necessitating efforts to re-educate new staff.
• Cumbersome referral processes. The referral processes for some SSPA programs were complicated and/or new. This meant that the referring agencies were sometimes slow to start making referrals or were reluctant to refer into the program.
• Highly mobile families. After initial referral, some families were difficult to enroll because of their residential mobility. Some of this mobility was undoubtedly related to their violence exposure. This prevented the programs from completing the intake process and/or determining eligibility for referred families. The mobility of some families also made completing therapy sessions difficult.
• Families' negative views or experiences with mental health services referrals. Some referral sources faced problems with families' negative experiences or perceptions about mental health services, and sometimes around issues of mandatory reporting of child abuse and a desire to keep violence exposure private. These factors affected families' willingness to accept a referral for mental health services.
• Complexity of families' treatment needs. Many SSPA programs found that the families had multiple and complex situations and difficulties meeting day-to-day needs, including safety issues. These immediate needs often took precedence for families and made it difficult for service providers to deliver the therapy component of the program.
• Close relationships with families. The referral sources and program staff at some SSPA programs invested time and resources into building close relationships with families. These close relationships helped the referral sources engage families in discussions about violence exposure and the SSPA program. The program staff felt that the closer relationships helped engage families in the intervention.
• Culturally inconsistent services. Some of the SSPA programs struggled with a lack of cultural consistency related to race/ethnicity, socioeconomic status, and other demographic characteristics between those providing services and the families being served. These differences posed challenges in service delivery if families could not be served by providers proficient in their culture or primary language.
• Clearly defined program parameters. Some SSPA programs had problems clarifying the roles and responsibilities of different program staff and determining when to end services for individual families. This confusion within the program sometimes made if difficult for the program to engage and retain families in the program.
• Flexible service delivery model. Some SSPA programs had flexibility within the service delivery model that allowed them to adjust caseloads, the intervention setting, or program staff roles. This flexibility helped the programs be responsive to families' circumstances.
• Staffing issues among service providers. Some of the service providers experienced issues with staff turnover, ongoing training needs, and staff burnout. These types of staffing issues made it challenging to maintain consistent, quality services.
• Close communication and coordination between program staff and other service providers. Some SSPA programs worked to establish and maintain clear lines of communication between program staff and other service providers working with the families. Coordination strategies, such as multidisciplinary or case coordination meetings, provided opportunities for sharing information, coordinating services and supports, and planning next steps.
• Strained relationships between child welfare and domestic violence agencies. Historically, collaboration between child welfare and domestic violence agencies has been difficult because of different priorities. Traditionally, child welfare agencies focus on the needs and safety of the child, whereas domestic violence agencies focus on the battered woman and her needs. Some of the SSPA programs found it challenging to navigate these divergent perspectives and integrate the different approached to families.

Table 3 summarizes these factors and describes the implications for designing programs and providing services to families.

Conclusions

Looking across the SSPA programs, it is clear that the only feature shared by the interventions was their goal: providing interventions for children exposed to violence. Otherwise, the programs varied across every dimension, complicating the challenge of evaluating the process of implementing them. Therefore, it is necessary to be cautious about the conclusions drawn from this assessment of the implementation processes.

Given these limitations, however, it is possible to detect broad patterns across the sites' implementation experiences. Despite successes in launching programs and delivering needed services to children exposed to violence, most programs faced difficulties getting referrals, engaging families in treatment, and providing a program that aligned with the families' individual priorities. We briefly discuss each of these points in more detail below.

For many reasons, most sites struggled with lower-than-expected referrals throughout implementation. Some programs experienced difficulties establishing or enhancing collaborative relationships with the partner agencies/organizations that were providing referrals into the program. In some cases, the agency's own services to the family took precedence over identifying and referring for violence exposure. In other situations, the research component of the intervention made referral sources reluctant to refer, knowing that some families would not receive the program. The structure of the referral process itself also provided challenges, with proximity and the burden on referral sources playing a role in the flow of referrals into the program. These challenges suggest that, when designing programs for children exposed to violence, it is important to develop strategies for educating, collaborating, and maintaining strong relationships with referral sources.

The SSPA programs also struggled with engaging the families in the interventions being offered. The challenges arose for a variety of reasons, including the multitude of stressors faced by families, perceived stigma of mental health treatment, cultural differences, and families' reluctance to participate in a research project. The experiences of the SSPA programs highlight the critical role that engagement and retention strategies play in providing community-based programs for children exposed to violence.

Finally, with multiple and complex treatment needs in addition to pressing basic needs, families of children exposed to violence may need interventions with flexibility in the array of services and in the sequencing of components, focusing first on addressing basic needs. Among the diversity of program setting, population, and intervention types, this may be the most consistent experience of the 15 SSPA interventions and therefore potentially the most central lesson for future undertakings focused on addressing the needs of children exposed to violence.

While we do not yet know the impact of the interventions on child and family outcomes, the findings from this process evaluation provide guidance for future community-based programs for children exposed the violence. The upcoming report on the outcome evaluation component of the national evaluation will address the questions related to the effect of the programs on child and family outcomes.

Reference

Finkelhor, D., R. Ormrod, H. Turner, and S. L. Hamby, “Violence, Abuse, and Crime Exposure in a National Sample of Children and Youth” Pediatrics, Vol. 124, No. 5, 2009, pp. 1411–1423.