Evaluation Design for the Department of the Air Force's True North Program: Development of a Logic Model and Program Measures

by Miriam Matthews, Thomas E. Trail, Samantha E. DiNicola, Coreen Farris, Jenna W. Kramer, Isabelle González

This Article

RAND Health Quarterly, 2024; 11(3):7

Abstract

Developing strong resiliency and care solutions for airmen and guardians is key to human capital development and force readiness. The True North program is one of the Department of the Air Force's (DAF's) most significant recent investments in promoting the resiliency of its people. Assessing the program's level of success, justifying funding, and informing decisions about the program's future will require a rigorous evaluation. The authors of this study (1) identify desired outcomes for members participating in the program, (2) define appropriate measures of effectiveness that could be used in evaluating the True North program, and (3) make recommendations for ongoing internal evaluation of the program. The True North program encompasses selected installation welcome centers, embedded religious support teams (RSTs), and embedded mental health (EMH) teams. To determine how DAF might evaluate this program and its components, the authors reviewed relevant policies and procedures and literature relevant to the program components and conducted interviews with 17 True North program personnel and 21 group and squadron commanders. They present a program logic model to determine potential evaluation measures.

For more information, see RAND RR-A1753-1 at https://www.rand.org/pubs/research_reports/RRA1753-1.html

Full Text

In support of a True North program evaluation, the RAND Corporation's Project AIR FORCE (PAF) was asked to (1) identify desired outcomes for members participating in the programs; (2) define appropriate measures of effectiveness that could be applied to the True North program; and (3) establish recommendations for ongoing internal evaluation.

Approach

The True North program encompasses selected installation welcome centers, embedded religious support teams (RSTs), and embedded mental health (EMH) teams. To determine how the Department of the Air Force (DAF) might evaluate this program and its components, PAF researchers reviewed relevant policies and procedures and literature relevant to the program components. We also conducted interviews with 17 True North program personnel and 21 group and squadron commanders. We developed a program logic model to determine potential evaluation measures.

Key Findings

Our research yielded the following key findings:

  • Research shows promising impacts of embedded behavioral health, including in military, education, and health care settings. These impacts include greater service use, increases in accuracy of diagnosis, and greater continuity of care.
  • Our interviews found that True North program personnel perceived positive outcomes from the program, but our research also showed that challenges with funding changes and delays in embedding personnel can limit program effectiveness.
  • Interviewees indicated that wing chaplains often provide helpful supplemental funding to RSTs, but disagreement exists about the level at which RSTs should be embedded.
  • Many locations within the True North program do not have True North welcome centers.
  • Critical elements of determining the effectiveness of the True North program will be assessing whether the program has received all requisite inputs and whether activities have been implemented as intended.

Recommendations

As a result of our research, we recommend the following:

  • Adopt and communicate the program logic models in ways that ensure that stakeholders know and understand them.
  • Continue to engage stakeholders in the evaluation design process.
  • Implement assessment tools that are aligned with the program logic model.
  • Present the results of evaluations in ways that are clear and useful for stakeholders.
  • Modify the program logic model, as needed, drawing from the results of evaluations.
  • If DAF Air Force Integrated Resilience considers creating or using a database to track True North measures, involve stakeholders in its development.

This research was prepared for the the Department of the Air Force and conducted within the Workforce, Development, and Health Program of RAND Project AIR FORCE.

RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.