Evaluation Design for the Department of the Air Force's True North Program: Development of a Logic Model and Program Measures

In support of a True North program evaluation, the RAND Corporation's Project AIR FORCE (PAF) was asked to (1) identify desired outcomes for members participating in the programs; (2) define appropriate measures of effectiveness that could be applied to the True North program; and (3) establish recommendations for ongoing internal evaluation.

Approach

The True North program encompasses selected installation welcome centers, embedded religious support teams (RSTs), and embedded mental health (EMH) teams. To determine how the Department of the Air Force (DAF) might evaluate this program and its components, PAF researchers reviewed relevant policies and procedures and literature relevant to the program components. We also conducted interviews with 17 True North program personnel and 21 group and squadron commanders. We developed a program logic model to determine potential evaluation measures.

Key Findings

Our research yielded the following key findings:

• Research shows promising impacts of embedded behavioral health, including in military, education, and health care settings. These impacts include greater service use, increases in accuracy of diagnosis, and greater continuity of care.
• Our interviews found that True North program personnel perceived positive outcomes from the program, but our research also showed that challenges with funding changes and delays in embedding personnel can limit program effectiveness.
• Interviewees indicated that wing chaplains often provide helpful supplemental funding to RSTs, but disagreement exists about the level at which RSTs should be embedded.
• Many locations within the True North program do not have True North welcome centers.
• Critical elements of determining the effectiveness of the True North program will be assessing whether the program has received all requisite inputs and whether activities have been implemented as intended.

Recommendations

As a result of our research, we recommend the following:

• Adopt and communicate the program logic models in ways that ensure that stakeholders know and understand them.
• Continue to engage stakeholders in the evaluation design process.
• Implement assessment tools that are aligned with the program logic model.
• Present the results of evaluations in ways that are clear and useful for stakeholders.
• Modify the program logic model, as needed, drawing from the results of evaluations.
• If DAF Air Force Integrated Resilience considers creating or using a database to track True North measures, involve stakeholders in its development.