Cover: Identifying Gaps and Areas for Improvement in the FEMA Qualification System for Incident Workforce Positions

Identifying Gaps and Areas for Improvement in the FEMA Qualification System for Incident Workforce Positions

Recommendations for Developing an Improvement and Evaluation Process

Published Mar 6, 2024

by Leslie Adrienne Payne, Andrea M. Abler, Susan G. Straus, Jason Michel Etchegaray

Download

Download eBook for Free

FormatFile SizeNotes
PDF file 0.9 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Purchase

Purchase Print Copy

 Format Price
Add to Cart Paperback94 pages $29.00

Research Questions

  1. What is currently working well in FQS?
  2. What are the current gaps and challenges in FQS?
  3. What are the potential solutions (i.e., course of action recommendations) for addressing those gaps and challenges?
  4. To what extent are metrics available to measure the effectiveness of five main FQS areas: Position Task Books, coaches and evaluators (C&Es), training, the Qualification Review Board, and the Deployment Tracking System?

To fulfill its mission objectives, the Federal Emergency Management Agency (FEMA) requires a well-trained workforce. The Post-Katrina Emergency Management Reform Act of 2006 mandated that FEMA develop a personnel credentialing process for incident workforce personnel responding to disaster situations. FEMA, in turn, created the FEMA Qualification System (FQS), a credentialing process that focuses on standardizing personnel abilities through training and applied experience.

Following concerns raised about the efficiency and effectiveness of FQS to provide the needed structure to qualify individuals for their assigned positions, FEMA's Integration Branch asked the authors to identify existing FQS gaps and areas for improvement and examine ways to measure and monitor the effectiveness of FQS in the future. This report details the authors’ approach to addressing both issues and the related findings so that FEMA can ensure the development of employees who provide vital disaster recovery services to the United States and its people.

Key Findings

  • Overall, FQS has a fair number of strengths, and stakeholders see value in the concept behind FQS and its component parts.
  • In practice, the component pieces do not always function as designed.
  • There are very few systematic measures and evaluation activities to gauge whether processes in FQS components are implemented effectively and in ways that align with FEMA's goals.

Recommendations

  • Develop a standardized checklist consisting of required Position Task Book elements and a checklist that documents when tasks from Position Task Books and content from training are aligned.
  • Analyze C&E ratings for common rating errors and for possible differences in ratings by protected class or other factors.
  • Institute supervisor oversight and conduct performance review of C&Es.
  • If none currently exists, establish a grievance/redress process for trainees who feel unfairly evaluated by C&Es.
  • For Kirkpatrick Level 2 evaluations, create pre-tests and post-tests to assess improvement in learning.
  • For Level 2 evaluations, analyze the psychometric properties of tests, ideally using item response theory.
  • Consider using a quantitative model that combines numeric ratings on inputs currently used by the Qualification Review Board to make qualification recommendations.
  • Consider an alternate Qualification Review Board model that better addresses the technical nuances of each cadre and job position. For example, this could take the form of convening, for each cadre, a technical review panel with specialists specific to that cadre.
  • Determine the frequency with which users select Incident Workforce personnel for deployment by name rather than relying on the Deployment Tracking System algorithm, and conduct a qualitative analysis of justifications.

This research was sponsored by FEMA's Integration Branch and conducted in the Disaster Management and Resilience Program of the RAND Homeland Security Research Division.

This report is part of the RAND research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.