The Welcome Baby Program: An Implementation and Outcomes Evaluation

First 5 LA (F5LA) is a public entity that supports collaborative work across Los Angeles County to strengthen families, communities, and systems of services and supports so that all children in the county are born healthy, maintain a healthy weight, are safe from abuse and neglect, and enter kindergarten ready to succeed in school and life (First 5 LA, 2014a). Over the last decade, F5LA has developed a unique approach to supporting families through a tiered home visiting approach. Among women living in 14 designated communities in Los Angeles County, births at participating hospitals are screened for risk; the highest-risk families are offered one of two intensive evidence-based home visiting programs. Lower-risk families are offered Welcome Baby, which provides a visit in the hospital after birth and up to five post-partum visits through the child's ninth month of life. The combination of the program's “light touch” and its targeting of low- or moderate-risk families sets Welcome Baby apart from many home visiting programs, which tend to be more intensive and target high-risk families (Michaloupolos et al., 2015). With the recent addition of Family Connects (a home visiting program providing only one to three visits) to the federal evidence-based home visiting list (Sama-Miller et al., 2017), there is considerable interest in the potential of less-intensive programs to improve family outcomes at less expense than other federally listed programs.

This article presents the findings from our evaluation of the Welcome Baby program. F5LA has undertaken a series of evaluations of Welcome Baby over the program's lifecycle, and this evaluation examines the realization of implementation and outcome goals. The evaluation focused on answering the following questions about the implementation and outcomes of Welcome Baby:

1. To what extent are sites implementing Welcome Baby to fidelity?
2. Is there variability in sites' ability to reach fidelity to Welcome Baby? If so, what factors account for this variability?
3. How are sites maintaining community resource and referral networks? What if any gaps exist in these networks?
4. What are participant perceptions of and experiences with the program and Welcome Baby service providers?
5. What factors contribute to participants leaving the program early?
6. To what extent do participants achieve short- and intermediate-term outcomes?
7. What are the relationships between program fidelity outcomes and participant outcomes?

To address these questions, we examined the fidelity domains in the Welcome Baby fidelity framework and assessed outcomes drawn from the stated goals and objectives in F5LA's Welcome Baby logic model (First 5 LA, 2014b). We used multiple methods to analyze data from the first 12 sites that implemented the program,¹ including interviews with program staff, focus groups with program participants, staff and participant surveys, program administrative data, and document review. These data were collected during a period spanning January 2016 to December 2017.

This evaluation was designed to help F5LA decide how to best allocate resources for training, program monitoring, and other program management activities. The evaluation findings are also likely to be valuable to other home visiting programs, as the fidelity components identified in Welcome Baby are similar to those of other family services. Finally, the information in this study will contribute to the ongoing discussion in the home visiting field regarding the use of universal, and lower-intensity, home visiting services.

The Welcome Baby Program

During pregnancy or soon after delivery, a Welcome Baby staff member conducts a risk assessment with families interested in receiving home visitation services. The triage process involves using a screening tool to assess a family's risk level and determining whether the family resides in one of F5LA's focal communities. Families in the focal communities who are at high risk are recommended to area home visiting programs that provide more intensive home visiting services than Welcome Baby. Families in the focal communities who score under high risk on the screening tool are offered Welcome Baby. (The focal communities are among the highest-risk communities in the county, so even lower-risk families may face many child rearing challenges.)

Families enrolled in Welcome Baby may receive three prenatal home visits, a visit in the hospital after birth, and up to five postpartum home visits. This evaluation focuses on the postpartum visits (see Figure 1). Each Welcome Baby home visit includes risk and developmental assessments, and these are used to guide the topics covered during the visit, as well as any referrals made to other services. The Welcome Baby curriculum delivered at each engagement point is designed to be flexible to meet families' differing needs. Welcome Baby emphasizes that a service relationship characterized by connectedness, empathy, and empowerment helps promote positive behavior changes (Edelman, 2004). During the home visits, parents receive support and information on such topics as postpartum depression, breastfeeding, immunizations and well-baby doctor visits, and home safety. During each home visit, home visitors model reflective communication and empathy to teach parents interaction skills that help them better connect with their child.

1. First or second trimester of pregnancy

   Visit in the home

2. Phone call check-in

3. Third trimester of pregnancy

   Visit in the home

4. Baby is born

   Postpartum hospital visit

5. Nurse home visit within one week of mom and baby coming home

6. Baby's 2–4 weeks

   Visit in the home

7. Baby's 2 months

   Visit in home

8. Baby's 3–4 months

   Visit in the home

9. Baby's 9 months

   Final visit in the home

All appointments are held with a personal parent coach or nurse who offers women support and education every step of the way.

SOURCE: LA Best Babies Network, “Welcome Baby Timeline of Program Visits,” fact sheet, undated.

F5LA developed a Welcome Baby Fidelity Framework (see Table 1) to guide the implementation of the program, and the framework specifies central components of the program in terms of staff training and expertise, number and timing of family encounters, and approach to interactions with families. Examining whether Welcome Baby sites meet these fidelity standards and whether the standards are related to client outcomes were central components of this evaluation. Each of the Welcome Baby positions has different qualification requirements, such as degrees, certifications (e.g., lactation consultant certification), as well as preferred experience and language capabilities. Each position also has training requirements, which consist primarily of training provided by Welcome Baby.

Data Sources

To address these evaluation questions, we used multiple data sources (e.g., interviews, focus groups, client surveys, staff surveys) and analytic methods (e.g., quantitative analysis of both primary and secondary data, qualitative analysis of interview data). Table 2 provides an overview of the data sources and relationship to each evaluation question. Of note, we intended to include observational assessments of the home visits performed by home visitor supervisor staff in our analyses, but because of a delayed start in the use of the observational assessment tool during our study period, too few observation assessments were completed to include in our analyses.

Findings

We tailored the methods for addressing each evaluation question using the multiple data sources and qualitative and quantitative methods as appropriate. For the fidelity analysis, we identified a threshold that we could use to assess whether or not a site had achieved the fidelity standard. We briefly summarize the main findings for each evaluation question.

1. To What Extent Are Sites Implementing Welcome Baby to Fidelity?

There was great variability in the degree to which the sites achieved fidelity to the Welcome Baby model. For each site, we assessed fidelity for the 11 of the 14 fidelity domains for which we had data. No site achieved fidelity in all 11 domains, but every site achieved fidelity in supervisory requirements and participant perceptions of the relationship. Individual sites achieved fidelity in 18 to 80 percent of measured domains, with an average of 48 percent.

2. Is There Variability in Sites' Ability to Reach Fidelity to Welcome Baby? If So, What Factors Account for This Variability?

For each of the 11 fidelity domains, there were large differences in the proportion of sites that met the fidelity threshold. For two domains (supervisory requirements and participant perceptions of the relationship), all assessed sites achieved fidelity thresholds; for another three domains (staff qualifications, reflective supervision, and hospital enrollment), only one site achieved fidelity thresholds. For the other domains, between five and eight sites achieved fidelity thresholds. Across all of the domains, an average of 5.5 sites achieved domain fidelity. In several domains, sites also varied in achieving thresholds by specific elements of the domain criteria (e.g., by staff position, visit type). Findings from site interviews with Welcome Baby staff provided information that helps explain the challenges in meeting the fidelity thresholds.

3. How Are Sites Maintaining Community Resource and Referral Networks? What, If Any, Gaps Exist in These Networks?

The completeness of referral directories varied widely across sites. Very few sites had developed the organizational infrastructure to facilitate successful referrals (e.g., memoranda of understanding with service providers and referral forms). The Welcome Baby program provides sites with protocols that outline procedures for five referral types, including domestic violence, early intervention for child developmental delay, postpartum care, maternal depression, and suicide prevention. These five Welcome Baby protocols require monitoring referral completion; four require that Welcome Baby staff help clients gain access to the services, and two emphasize client confidentiality. Staff across all sites reported referring to a wide range of referral resources, regardless of the infrastructure developed to facilitate successful referrals.

4. What Are Participant Perceptions of and Experiences with the Program and Welcome Baby Service Providers?

Both qualitative and quantitative data indicated that Welcome Baby participants generally had a positive perception of the program. In focus groups, participants overwhelmingly reported that the Welcome Baby program met their needs and helped them connect with services. Program participants indicated that they would participate in the program again if seeking parenting help in the future and rated their relationships with the Parent Coaches extremely positively. Clients noted that the program enrollment process was easy, and they appreciated most the assistance with breastfeeding. Overall, Welcome Baby staff were perceived as responsive to their needs, easy to communicate with, accessible, and flexible. Clients identified a few areas for improvement, such as offering more visits and providing program materials in non-paper formats. Survey results corroborated the information from the focus groups: Across sites and time (i.e., from the two-to-four-week visit to the nine-month visit), program participants generally had positive perceptions of the Welcome Baby program and their relationship with their Parent Coaches.

5. What Factors Contribute to Participants Leaving the Program Early?

We assessed whether participant characteristics and fidelity during a previous Welcome Baby program visit predicted families getting a successive visit. Family characteristics associated with being at risk of poor outcomes (e.g., a high risk score, being young) were often associated with a greater likelihood of transitioning from the hospital to a registered nurse (RN) visit, but a lower likelihood of staying in the program for later visits. For the two-to-four-week visit and later visits, adherence to Welcome Baby fidelity standards, such as covering the curriculum, was related to lower rates of participants leaving the program.

6. To What Extent Do Participants Achieve Short- and Intermediate-Term Outcomes?

We examined Welcome Baby participants' outcomes across all sites and visits for 12 outcome measures. Where regional or national benchmarks were available, Welcome Baby participants exhibited better outcomes in more than half of the outcome areas measured, including more positive parenting practices, higher levels of breastfeeding, and safer sleep environments (i.e., back sleeping and no co-sleeping) than benchmarks. Welcome Baby participants exhibited lower levels of family planning and exclusive breastfeeding compared to the benchmarks.

7. What Are the Relationships Between Program Fidelity Outcomes and Participant Outcomes?

We estimated the relationship between the 12 outcomes and eight fidelity measures. We found little evidence of relationships between program fidelity and participant outcomes. Across the eight fidelity components included in this analysis, the ones most likely to be associated with improved outcomes were staff qualifications, staff training, reflective supervision, home visitor workload, and curriculum content coverage.

The findings from this evaluation provide Welcome Baby stakeholders with data-based information on the implementation and outcomes of the program. Additionally, the findings inform the home visiting field as a whole, providing rigorous analysis of such issues as factors contributing to home visiting program attrition and assessing referral networks.

Conclusions and Recommendations

Overall, we found that the Welcome Baby program sites were meeting many of the program's implementation and outcome goals and that participants had favorable views of the program. The evaluation findings highlighted some areas where F5LA could explore improvements or clarity in policies and procedures. Next, we provide suggestions for these areas.

Staff Qualifications and Training

Sites typically did not perform well in the staff qualifications and training fidelity domains, although some sites performed well in one or the other. A review of staff qualifications and training requirements for each position might be helpful to see how relevant these are in the future recruitment and training of Welcome Baby program staff. These two fidelity domains are particularly critical because they appear to be related to program attrition; sites that met the fidelity thresholds appear to have had less participant attrition than those that did not.

Supervision

While sites generally achieved fidelity in terms of supervision caseload levels, the frequency and quality of reflective supervision may benefit from further examination. Some staff positions reported not receiving reflective supervision very frequently, and staff across positions and sites questioned reflective supervision's quality and value. This is particularly important because reflective supervision was related to several positive participant outcomes, such as well-child visits, immunizations, home safety, and safe sleeping practices. Reflective supervision also appears to negatively relate to attrition between the hospital and RN visit. F5LA may want to further examine the quality of the reflective supervision, especially for hospital liaisons and RNs.

Hospital Enrollment

This fidelity domain consisted of two components: (1) approaching 90 percent of eligible families in the hospital and (2) enrolling 40 percent of those approached. Most sites missed the 90-percent approach target. Since sites reported challenges meeting the target for approaching eligible families in the hospital, F5LA may want to address staff coverage issues (e.g., not having staff available 24/7, including nights and weekends). While most sites met the target for enrolling families, there was wide variation, suggesting that staff across sites could learn from one another on effective enrollment practices. F5LA may also want to consider whether site-specific targets may be more appropriate, given the number of births and Welcome Baby staffing levels at the different participating hospitals. Now that more data have been collected on Welcome Baby, the threshold enrollment rate can be adjusted based on observed site enrollment rates to set more realistic targets.

Service Dosage

Sites exhibited wide variation in the percentage of participants who received four or more postpartum Welcome Baby visits. Although program staff thought that the large gaps between the three-to-four month and nine-month visits contributed to attrition, the analysis found that most attrition occurred between the hospital and RN visit. F5LA may want to undertake continuous quality improvement approaches to increase program retention between the hospital enrollment visit and the first in-home visit. Staff at higher-performing sites may have lessons learned that could improve performance at other sites.

Home Visit Content

Overall, across sites and across visits, coverage of the Welcome Baby curriculum was good. Since coverage was lower at the hospital visit, F5LA may want to examine whether crucial content is being missed in the hospital.

Community Referral Process

Performance on this fidelity domain was extremely varied across sites, representing another potential opportunity to learn from best practices at some sites. Due to the limitations of the Stronger Families Database (SFDB), the administrative database maintained on the Welcome Baby program, we were able to evaluate this domain for only a short time period and did not include it in our analyses of factors related to program attrition and outcomes. Based on review of each site's documentation on the referral process, F5LA should consider the development of detailed protocols for all high-priority referral types, including public benefits, alcohol, smoking, and drug treatment (all protocols should include the provision of client confidentiality). F5LA should also support sites in developing and maintaining a standardized referral directory and establish memoranda of understanding with service providers to improve service access.

Outcomes

Sites' abilities to achieve outcomes, such as family planning, exclusive breastfeeding, and co-sleeping, varied widely and, therefore, could be areas to target for staff retrainings and booster sessions. Sites that performed better may be able to share experiences and lessons learned with sites that did not perform as well. Rates of postpartum depressive symptoms were extremely low in comparison to other benchmarks, suggesting that administration of the postpartum depression screener may be improved.