The National Guard Youth ChalleNGe program is a residential, quasi-military program for youth ages 16 to 18 who are experiencing difficulties in traditional high school. This capstone report describes some of RAND's research and analyses on the ChalleNGe program from late 2017 through mid-2020. The authors focus on identifying and recommending strategies for programs to measure participants' long-term outcomes.
Developing Outcome Measures for the National Guard Youth ChalleNGe Program
- How can ChalleNGe's long-term outcomes be measured in a way that demonstrates the extent to which the program is meeting its mission?
- Are current data collection methods sufficient to support measurement of long-term outcomes?
The National Guard Youth ChalleNGe program is a residential, quasi-military program for youth ages 16 to 18 who are experiencing difficulties in traditional high school. The program provides classroom instruction and other structured activities to young people at 40 different sites. This capstone report describes some of RAND's research and analyses on the ChalleNGe program from late 2017 through mid-2020.
Using a mixed-methods approach, the RAND team developed a program logic model, carried out a series of site visits, and planned and executed numerous analytic efforts. Many of the results of specific analytic efforts are described in other reports; here, the authors focus on identifying and recommending strategies for programs to measure participants' long-term outcomes. Consistently measuring long-term outcomes will allow the program to determine overall progress toward meeting its mission of ensuring that program participants are prepared for success as productive citizens.
ChalleNGe sites collect considerable data, but postgraduation placement data are lacking
- ChalleNGe sites have detailed information on the number of applicants, entrants, and graduates, as well as on other relevant outcomes, such as the amount of community service performed and standardized test scores.
- Sites gather placement data through mentors, and such data generally include no more than 12 months of minimal information; sites frequently express concern that mentor reporting drops off sharply after graduation. Sites may not be able to document industry or occupation, existence of benefits, or even total earnings.
- The data collected by ChalleNGe sites provide useful information and meet reporting requirements, but this information is not sufficient to determine how well the program is meeting its stated mission of creating long-term change and helping young people go on to become productive citizens.
ChalleNGe does not keep a single, authoritative administrative database
- Many of the sites use a similar database, the Cadet Tracker.
- RAND researchers have collected consistent data from individual sites for the past four years, and the National Guard Bureau also collects data periodically.
- Some sites have tested other methods of data collection; these include developing a customized database, surveying graduates, and focusing additional resources on tracking graduate placement.
- Each site should select the data collection strategy or strategies most appropriate to the site context.
- All sites should discuss the importance of data collection with cadets and parents.
- Site and program staff should budget for data collection.
- All sites should collect, at a minimum, data on graduates' educational attainment, labor force experience, and progress on other core components.
- Sites should collect data for three years postgraduation.
- The ChalleNGe program should invest in a single, consistent administrative database.
Table of Contents
ChalleNGe: Program Design and Logic Model
Existing Data and Approaches to Program Measurement
Approaches for Measuring Long-Term Outcomes and Determining Program Effectiveness
Conclusions and Recommendations
Contextual Factors That Influence ChalleNGe Sites
How Similar Youth Programs Approach Data Collection and Outcomes Measurement
Potential Measures of Citizenship, Leadership/Followership, and Service to Community
Survey of Washington Youth Academy Graduates