Process Evaluation of the New Mexico Maternal, Infant, and Early Childhood Home Visiting Competitive Development Grant

by Matthew Chinman, Sarah B. Hunter, Jill S. Cannon, M. Rebecca Kilburn, Melody Harvey, Mollie Rudnick

This Article

RAND Health Quarterly, 2014; 4(3):3

Abstract

This article describes the evaluation of the New Mexico Home Visiting Competitive Development Grant, which sought to pilot test the use of implementation supports to improve the development and implementation of home visiting programs. Each community was to use Getting To Outcomes® (GTO) and ECHO® (Extension for Community Healthcare Outcomes) to support their work. The GTO framework promotes capacity for high–quality programming by specifying ten steps that practitioners should take and by providing support to complete those steps. ECHO involves specialists providing training and technical assistance via distance technology to community practitioners in rural areas to improve the quality of services. The grant was delayed, and the project's scope shifted significantly from the original plan. The evaluation documents significant challenges in meeting grant goals. A local team hired to facilitate GTO did not use it as designed, and no communities were trained in GTO. The coalitions that were developed operated with few resources or accountability, and made little progress on plans to enhance services. Only two of the four communities started home visiting after nearly a year and a half. ECHO was used with fidelity with those two programs. There was no change in the continuum of services to support children and families. Due to delays and lack of implementation of the proposed intervention, the evaluation was not able to assess the project's impact on child or family outcomes, nor did the project serve as a robust pilot test of the use of GTO and ECHO to improve home visiting implementation.

For more information, see RAND RR-639-REC at https://www.rand.org/pubs/research_reports/RR639.html

Full Text

New Mexico's children have ranked near the bottom or last in the annual KIDS COUNT state-by-state rankings of child well-being for over a decade (New Mexico Voices for Children, 2014), prompting interest in strategies that could improve child outcomes in the state. One strategy shown to be effective in improving child well-being is home visiting programs, which, according to the State of New Mexico's Children, Youth and Families Department (CYFD), deliver “informational, educational, developmental, referral/linkage, screening/evaluation, and other direct intervention and support services for families” (CYFD, undated, pg. 5). The State recognizes the promise of home visiting, but also that community organizations need additional assistance to implement them well—not just funding. To help improve implementation of home visiting programs, the New Mexico CYFD secured a federal competitive demonstration grant from the Maternal, Infant, and Early Childhood Home Visiting program. The resulting Home Visiting Competitive Development Grant (HVCDG) sought to improve the lives of children and families in a select group of high-need communities by building capacity—i.e., the knowledge and skills needed to complete key tasks that make programs successful—for home visiting program implementation through the Getting To Outcomes® (GTO) framework and ECHO® (Extension for Community Healthcare Outcomes) distance-learning approach. The GTO framework promotes community capacity for high-quality programming by specifying ten steps community practitioners should take and providing implementation support to complete those steps. ECHO involves specialists providing training and technical assistance via distance technology to community practitioners in rural areas to improve the quality of services for complex diseases.

The State asked the RAND Corporation, a codeveloper of GTO, to:

  • develop a GTO manual specifically tailored to home visiting
  • provide consultation (called “supervision” in GTO projects) to a local subcontractor (called the GTO facilitation team) on facilitating the use of GTO in the participating communities, and
  • evaluate whether the HVCDG project improved community capacity and child and family outcomes.

The State also subcontracted with the Center for Development and Disability at the University of New Mexico (CDD UNM) to adapt its ECHO telehealth approach to provide training and technical assistance (T/TA) to the home visiting programs in the participating communities (Luna, Quay, & McKinley counties, and the South Valley neighborhood of Albuquerque).

The State modified the HVCDG's timeline, sites, and scope significantly from the original proposal. First, the HVCDG started eight months late (in May 2012) because of state contracting delays. CYFD requested an extension from MIECHV; however, RAND had completed a large amount of the data collection before the State received the extension. Hence, this study covers the period of the originally planned activities, from September 2011 through November 15, 2013. Second, the HVCDG deleted its planned fifth community site (Grant County) when a more recent needs assessment showed less severe needs there, and replaced it with a “state level” unit of activity. Third, the State believed that additional improvements in other early childhood services were needed to maximize the value of investments in home visiting. Finally, the state shifted the GTO intervention from supporting the newly funded home visiting programs as originally planned to the community coalitions. As a result of these changes, HVCDG adopted the following objectives:

  1. help sites form effective early childhood coalitions (facilitated by the GTO facilitation team)
  2. enhance the continuum of services needed to successfully support families
  3. improve the infrastructure of the sites to deliver home visiting.

Methods

RAND conducted a process evaluation to assess: coalition development and activities, change in the continuum of early child care and related services, capacity of the home visiting programs, and use of GTO and ECHO (a planned outcome evaluation was eliminated after the change in scope). We developed a research question for each HVCDG objective, and added a fourth research question to evaluate how the HVCDG used GTO and ECHO (see Table 1). To answer the four research questions, we reviewed documents, interviewed stakeholders, rated the quality of the coalitions' plans, and analyzed data from the newly started home visiting service programs.

Document Review

To evaluate GTO utilization, we reviewed documents from the GTO consultation and facilitation teams. We also used documents from the coalitions to document their fidelity to the GTO framework using the GTO Activity Monitoring Tool that lists all of the subtasks prescribed in GTO's ten steps. We also determined the number of coalition meetings held, the meeting content, and the number and diversity of participants. Finally, we reviewed the T/TA tracking data maintained by CDD UNM for measuring ECHO implementation.

Table 1. Overview of Research Questions and Methods

Research Questions Measure/Data Collection Tool Sources Data Collection Time points
1. Did the four participating communities form early childhood coalitions and begin to implement requisite activities?

Document abstraction (e.g., coalition meeting minutes)

Interviews with community stakeholders

Plan Quality Index

Kansas University community coalition building tool

Coalition documents

Coalition members

Community Action Plans

Beginning of community organizing; Fall 2013
2. To what extent did the project utilize GTO and ECHO distance learning to support the work of the coalitions and home visiting programs?

Document abstraction (e.g., coalition meeting minutes, ECHO T/TA database, GTO facilitation tracking log)

Interviews with community stakeholders, GTO consultation team and GTO facilitators, ECHO T/TA provider

Coalitions, GTO facilitators, and ECHO T/TA provider

Coalition members, GTO consultation team and GTO facilitators, ECHO T/TA provider

Fall 2013
3. Did the participating communities enhance the continuum of services they need to support families?

Document abstraction (e.g., GTO facilitation tracking log)

Interviews with community stakeholders, state-level stakeholders; GTO facilitators, ECHO T/TA provider

Continuum of services list

GTO facilitators, and ECHO T/TA provider

Coalition members, GTO facilitators, ECHO T/TA provider, state level stakeholders

Beginning of community organizing; Fall 2013
4. Did the participating communities improve their infrastructure for home visiting services?

Home visiting administrative data

GTO Capacity Interview

Home visiting programs

Home visiting program leaders

Fall 2013

Interviews

We interviewed both the GTO facilitation team and the CDD UNM T/TA provider as a data source to document GTO and ECHO utilization. We also interviewed three to four community stakeholders from each site about HVCDG support (using GTO and ECHO), carrying out coalition-building activities, and the perceived change over about a year's time in the availability and helpfulness of the continuum of services to support families. We conducted additional interviews with stakeholders familiar with state early childhood policies regarding changes that may have occurred as a result of the HVCDG. Finally, we interviewed home visiting staff using the “GTO Capacity Interview” protocol, documenting the extent to which the program staff carried out key activities that previous studies have shown are associated with high quality programming (Livet and Wandersman, 2005).

Plan Quality Index

We rated the quality of the plans produced by the four community coalitions using the Plan Quality Index developed by Butterfoss, Goodman, et al. (1996). This instrument rates planning activities that GTO prescribes.

Home Visiting Service Delivery

As one indicator of how well each site established a new home visiting infrastructure, we analyzed data on the number of home visitors hired; families served; home visits made; and current families enrolled for the two communities that had started home visiting programs within the evaluation time period.

Findings

Research Question 1. Did the Four Participating Communities Form Early Childhood Coalitions and Begin to Implement Requisite Coalition Activities?

Although most coalition members were satisfied with their involvement and planned to stay involved, the coalitions did not carry out most activities needed to establish a strong structure during the 18-month evaluation period. Documents and interviews show that Quay, Luna, and McKinley county coalitions conducted most needs assessment and planning activities. However, the low Plan Quality Index ratings of all the coalitions suggest those actions did not result in high-quality plans (a key midstream product for the community coalitions). South Valley, whose coalition was mostly represented by a single organization, carried out far fewer coalition activities. None of the community sites engaged in any significant evaluation and quality improvement activities. Stakeholders across the communities noted that a lack of funding or other resources could undermine their coalitions' sustainability.

Research Question 2. To What Extent Did the Project Utilize GTO and ECHO Distance Learning to Support the Work of the Coalitions and Home Visiting Programs?

GTO was not well implemented with the four community coalitions. The GTO facilitation team made use of much less GTO training and consultation offered by RAND than is typical in GTO projects (Chinman, Hunter, et al., 2008; Acosta, et al., 2013). Program documents and interviews suggest that the GTO facilitation team did not fully implement GTO in the participating sites. While the GTO facilitation team presented the concept of the GTO framework to Luna and Quay county coalitions, they did not train any coalition members in GTO—and the GTO facilitation team carried out what actions were taken, instead of empowering the coalitions to take the lead in completing the GTO tasks. (Active participation is a key way to build capacity in the GTO framework.) The RAND team did not have administrative authority over the GTO facilitation team typical in GTO projects, and could only provide feedback to the team and the State. The State, which did have authority, did not hold the GTO facilitation team accountable—i.e., ask them to change their actions and implement GTO with fidelity the model.

Accordingly, the coalitions conducted few GTO activities in 18 months, although coalitions in other GTO projects have completed almost all the prescribed GTO activities in half that time. This may be because the coalitions in other GTO or similar projects were better resourced. Although the coalitions in HVCDG had similar amounts of facilitation support as other GTO projects as measured by the level of effort of the GTO facilitation team, they had no resources to apply to the management of the coalition itself (all members were voluntary). The activities that the coalitions completed most often were conducting needs assessments and developing goals and concrete benchmarks. But there was little evidence that the communities significantly engaged in the other activities related to the GTO steps during the HVCDG, and none of ten steps were addressed completely.

In contrast, specific activities carried out by the CDD UNM T/TA provider—a mix of didactic training, case consultations, and general Q&A for home visiting staff—appears to match that of past ECHO projects. The T/TA provider used a combination of onsite meetings and distance communications to help train home visiting program staff in Luna and Quay counties and provided T/TA to improve the delivery of their home visiting services. The level of contact between the T/TA provider and the two home visiting programs—3.5 times per month in Luna and 3.0 times per month in Quay—was close to the weekly contact prescribed by the ECHO model.

Research Question 3. Did the Sites Enhance the Continuum of Services They Need to Support Families?

Community stakeholders reported no improvement in the continuum of services. Community respondents also reported that the continuum of services is inadequate to serve families and several key services lack funding. Community stakeholders said the lack of awareness or understanding of available services can pose a barrier to accessing existing services, and coordination across services is needed. In fact, many community stakeholders indicated that the main value of the HVCDG was to promote awareness of community services related to young children. Also, HVCDG's plan called for the GTO facilitation team to engage in state-level discussions to create policies that would improve the continuum of services across the state (i.e., the state was the fifth “site”). However, the GTO facilitation team engaged in very few meetings with state-level officials and stakeholders. State-level interview respondents said there was a need for state-level discussions and that more meetings would have been helpful.

Research Question 4. Did the Sites Improve Their Infrastructure for Home Visiting Services?

Infrastructure is defined as the degree to which program staff are able carry out the many tasks known to be associated with effective programs, and these are prescribed by the GTO framework (Livet and Wandersman, 2005). We assessed the infrastructure of the home visiting programs by documenting the amount of services delivered and via ratings of responses from home visiting program leaders to the GTO Capacity Interview. Only two sites started home visiting services during the evaluation period (Luna, Quay), so those were the only programs interviewed and monitored. Staff from the Luna County program were rated as having moderate capacity, showing particular strengths in how they developed program goals and objectives, chose and planned the home visiting program, and steps it took to sustain services. Quay County was rated lower across all GTO domains, but did show some capacity for program planning. Both programs were rated low in their capacity to perform program evaluation and continuous quality improvement. Luna County met its enrollment target, Quay County did not. Interviewees there suggested that administrative delays by their fiscal agent contributed to the delayed program start.

Conclusions

The coalitions' weak structure and lack of resources and accountability limited their planning and impact on the continuum of services. The coalitions that were started by the GTO facilitation team did not possess sufficient resources, strong leadership, formalized structures, or a membership that actively participated in the requisite activities of coalition formation and implementation. Research on coalitions shows that weak organizational structure limits effectiveness (Zakocs and Edwards, 2006). This was, in large part, because HVCDG funding, except for the GTO facilitation team's time, was not available to support a more formalized infrastructure of the coalitions, but was instead intended to support home visiting programming. The members of the coalitions were participating on donated time. Given this, there was no management and staffing support to build the accountability needed for successful progress through the GTO steps. The Community Action Plan did not specify any budget information associated with the planned objectives. Finally, there was no strong accountability for the work of the coalitions. Although the GTO facilitation team worked with the coalitions, it did not have authority to hold the coalitions accountable for their work, and the coalitions had no benchmarks they were responsible to meet.

GTO was not implemented according to design. Applying GTO to home visiting was an innovation of the HVCDG, but did not occur because the GTO framework was applied only to the coalitions and not well implemented. This was because the GTO facilitation team participated in less training, consultation, and tracking than in past GTO projects; did not conduct GTO training with coalition member or home visiting staff, and did not empower the coalition members to take on any of the GTO-related tasks, an important feature that promotes capacity building. The GTO tasks that were completed—mostly elements of needs and resources assessments (GTO Step 1) and setting community goals and objectives (Step 2)—were largely done by the GTO facilitation team and shared with the coalition for their input.

The sites made little progress through the GTO steps due to loose organization along with a lack of staff support, budget, and other inputs, including few supports from the GTO facilitators. After two years, the GTO facilitation team, along with the members from the coalitions, did engage in some elements of the early GTO steps, but did not complete these or subsequent steps that related to home visiting. In past GTO projects that involved coalitions and programs with stronger organizational structures, and more GTO consultation, training and facilitation, practitioners were able to complete most or all of the ten steps in nine months or less (Chinman, Tremain, et al., 2009; Chinman, Acosta, et al., forthcoming). Although there was sufficient staffing available for GTO facilitation, the lack of management and support for individuals' participation in the coalitions remains an important barrier for the HVCD grant.

The coalitions, home visiting programs, and GTO facilitation lacked accountability. The coalitions, the home visiting programs, and the GTO facilitation team had few deliverables linked to a timeline that could ensure adequate progress. For example, the proposed timeline was to establish home visiting programs in the select communities within the first year of the award. Although this did not occur, there are other interim benchmarks that could have been established to monitor progress, allow feedback, and create an opportunity for midcourse corrections. Benchmarks alone do not guarantee accountability, but the lack of benchmarks makes accountability unlikely. The facilitation team did not carry out GTO according to its design during the evaluation period and the State did not hold them accountable.

The distance-learning T/TA was delivered mostly according to the ECHO model, but delays in HVCDG meant that a full pilot test of ECHO for home visiting was not completed. The T/TA provider for the HVCDG has been providing services (training, case consultations, Q&A sessions) that align with past ECHO demonstrations (Colleran et al., 2012; Arora, Thornton, et al., 2011). Yet the delays in implementation meant the HVCDG did not yield data for judging ECHO's impact on home visiting outcomes, preventing the pilot test of the use of ECHO.

Revisiting the project logic model demonstrates significant challenges to overcome. Reviewing the HVCDG project logic model demonstrates how the lack of certain activities (state level meetings, effective community organizing, and GTO training and facilitation) made it more difficult to achieve downstream outputs (e.g., increased capacity), short-term outcomes (strong coalitions, detailed community plans), and medium outcomes (enhanced continuum of services), which in turn will make it more difficult to achieve the long term outcomes of improved early childhood outcomes.

Limitations

The current evaluation has limitations that should be noted. First, it was beyond the scope of this evaluation to conduct a census of available services, and the use of key leaders as interviewees may have led to certain services being overlooked. Second, as stated in the Introduction, we did not evaluate the outcomes of children and families served by the newly created home visiting services as outlined in the proposal because no home visits had been delivered by the end of the first year of the project. Third, we had to rely on documentation about coalition and other community activities provided by the GTO facilitation team and CDD UNM T/TA provider. In some cases, this information is incomplete or missing (e.g., meeting minutes), and it is difficult to independently verify the accuracy of the information. We augment with interview notes where we can, but this is not always possible. Finally, RAND's role as both the provider of GTO supports (written material, training, T/TA) and the project evaluator may appear to compromise the independence of the evaluation. However, we took several steps to bolster the independence of the evaluation, including the use of different staff for data collection and GTO consultation; using multiple data collection staff to improve reliability, and adopting GTO implementation criteria from previous GTO research studies.

Policy Recommendations

We provide specific recommendations that can improve the HVCDG but are also applicable to similar projects planning to use coalitions and GTO.

Support Coalitions with Funding and Accountability

The coalitions in the HVCDG need more support—i.e., funding, paid staffing, and GTO facilitation—to better execute plans and manage the coalition (i.e., develop concrete roles for all participants, set benchmarks and timelines for accomplishing key tasks, and conduct outreach to expand the membership). Providing more support should also be accompanied with greater accountability, such as benchmarks for completing planned work. Future funding should be accompanied by requirements to document interim steps.

Implement GTO As It Was Designed

The effect that GTO may have on building home visiting program capacity can only be examined by actually providing GTO training and support to home visiting program staff. There may still be sufficient time and resources in the HVCDG to use GTO with home visiting programs as well as the coalitions.

Increase Accountability Across All HVCDG Project Activities

More accountability is needed in many aspects of the HVCDG to monitor interim steps and make changes. The State should: set performance benchmarks for the GTO facilitation team, the community coalitions, and the home visiting programs; monitor their activities; and establish consequences for both good and poor performance.

References

Acosta, J., M. Chinman, P. Ebener, P. S. Malone, S. Paddock, A. Phillips, P. Scales, and M. Slaughter, “An Intervention to Improve Program Implementation: Findings from a Two-Year Cluster Randomized Trial of Assets—Getting To Outcomes,” Implementation Science, Vol. 8, No. 87, 2013.

Arora, Sanjeev, Karla Thornton, Glen Murata, Paulina Deming, Summers Kalishman, Denise Dion, Brooke Parish, Thomas Burke, Wesley Pak, Jeffrey Dunkelberg, Martin Kistin, John Brown, Steven Jenkusky, Miriam Komaromy, and Clifford Qualls, “Outcomes of Treatment for Hepatitis C Virus Infection by Primary Care Providers,” New England Journal of Medicine, Vol. 364, 2011, pp. 2199–2207.

Butterfoss, F. D., R. M. Goodman, A. Wandersman, R. F. Valois, and M. J. Chinman, “The Plan Quality Index: An Empowerment Evaluation Tool for Measuring and Improving the Quality of Plans,” in D. M. Fetterman, S. J. Kaftarian, and A. Wandersman, eds., Empowerment Evaluation: Knowledge and Tools for Self-Assessment and Accountability, Thousand Oaks, Calif.: Sage, 1996, pp. 304–331.

Children, Youth and Families Department, State of New Mexico, “Affordable Care Act—Maternal, Infant and Early Childhood Home Visiting Program,” Announcement HRSA-11-179, undated. As of December 15, 2013:
http://cyfd.org/docs/fed_hv_comp_prop_ud.pdf

Chinman, M., J. Acosta, S. B. Hunter, and P. Ebener, “Getting To Outcomes®: Evidence of Empowerment Evaluation and Evaluation Capacity Building at Work,” in D. Fetterman, S. Kaftarian, and A. Wandersman, eds., Empowerment Evaluation: Knowledge and Tools for Self-Assessment, Evaluation Capacity Building and Accountability, Thousand Oaks, Calif.: Sage, forthcoming.

Chinman, M., S. Hunter, P. Ebener, S. Paddock, L. Stillman, P. Imm, and A. Wandersman, “The Getting To Outcomes Demonstration and Evaluation: An Illustration of the Prevention Support System,” American Journal of Community Psychology, Vol. 41, 2008, pp. 206–224.

Chinman, M., B. Tremain, P. Imm, and A. Wandersman, “Strengthening Prevention Performance using Technology: A Formative Evaluation of Interactive Getting To Outcomes,” The American Journal of Orthopsychiatry, Vol. 79, 2009, pp. 469–481.

Colleran, Kathleen, Erika Harding, Billie Jo Kipp, Andrea Zurawski, Barbara MacMillan, Lucie Jelinkova, Summers Kalishman, Denise Dion, Dara Som and Sanjeev Arora, “Building Capacity to Reduce Disparities in Diabetes Training Community Health Workers Using an Integrated Distance Learning Model,” The Diabetes Educator, Vol. 38, No. 3, May/June 2012.

CYFD—See Children, Youth and Families Department.

Livet, M., and A. Wandersman, “Organizational Functioning: Facilitating Effective Interventions and Increasing the Odds of Programming Success,” in D. M. Fetterman, and A. Wandersman, eds., Empowerment Evaluation in Practice, New York, N.Y.: Guilford, 2005, pp. 123–154.

New Mexico Voices for Children, 2013 Kids Count in New Mexico, Albuquerque, N.M.: KIDS COUNT, 2014. As of May 5, 2014:
http://www.nmvoices.org/wp-content/uploads/2014/01/NM-KC-2013-data-book-web.pdf

Zakocs, R. C., and E. M. Edwards, “What Explains Community Coalition Effectiveness? A Review of the Literature,” American Journal of Preventive Medicine, Vol. 30, No. 4, 2006, pp. 351–361.

The research described in this article was conducted in RAND Health and RAND Labor and Population, divisions of the RAND Corporation.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.