The Problem of Measuring Emergency Preparedness: The Need for Assessing ''Response Reliability'' as Part of Homeland Security Planning
Nov 14, 2008
Decisionmakers today largely assess emergency preparedness and homeland security "in the rearview mirror," looking at performance in actual events and responding to perceived failures. This policy brief frames a broad set of questions about how preparedness can be meaningfully evaluated and introduces the concept of response reliability, an alternative way of thinking about measuring preparedness that can answer the public's and policymakers' fundamental question: Will the systems we have put in place to respond to damaging events be able to deliver when called upon?
In the years since September 11, 2001, the question "is the United States sufficiently prepared for future natural disasters or terrorist attacks?" has been prominent in national policy debate. Looking at past response operations, it is generally easy to find things that did not go as well as expected or areas where planning and preparedness efforts seemed to fall short. Though learning from past experience is useful and important, managing preparedness planning and policy "in the rearview mirror"—constantly reacting to the perceived shortcomings of the last response and recovery operation—is not the right path to developing a homeland security and emergency-preparedness policy that serves the nation's needs.
Policymakers and the public need ways to prospectively assess preparedness so they know what they can expect when disaster strikes. Doing so is also critical for resource management. Because of the nature of emergency situations, there will almost always be "more that could have been done" when responding to a particular incident or disaster. But in a world of finite resources, achieving ideal performance in such situations will almost never be possible. Trying to address every shortfall identified retrospectively in response actions therefore risks either creating unsustainable demands for increasing preparedness expenditures or focusing scarce resources on shortfalls that may be easy to see but that may not be the most important preparedness problems.
Over the years, there have been many efforts to assess preparedness. Some have focused on evaluating the resources and activities that are easiest to quantify to provide some insight into what response systems will be able to accomplish. For example, we know that having the right equipment is important, so if equipment is not available to respond to types of incidents that concern us, then response operations are unlikely to go well. Other efforts have gone beyond inventorying resources to develop preparedness standards, assess less-tangible factors (such as training and leadership), or test plans and operations in exercises. Though they provide some insights into preparedness, these methods cannot answer the fundamental question of policymakers and the public: How certain should we as a nation be that the systems we have put in place to respond to damaging events will be able to deliver when called upon?
Whether there is a plan in place and whether supplies for assisting the victims of a disaster have been bought are key inputs to preparedness. However, those inputs will not produce the outcomes we want—actually providing relief to disaster victims when and where it is needed—if response organizations, infrastructures, and other components cannot deliver them. Confidence that response plans will be able to be executed as designed depends on the reliability of the system that is executing them. A plan to deliver supplies to an area that depends on one road and a single source of vehicles will be less likely to succeed than a plan that provides a variety of options and routes so operations can be adapted easily in response to the changing operational situation.
To address this concern about response performance, response reliability should be evaluated as part of preparedness measurement efforts. Such an assessment would be based on the nature of the system of response organizations, capabilities, and resources and the factors that shape how well it responds. The starting point is to map the system and identify the different elements that shape its performance. Assessing the reliability of the system then requires identifying what could go wrong, estimating the likelihood of breakdowns, identifying their impact on performance (e.g., would a particular failure mode completely disrupt response operations or just reduce their efficiency or effectiveness?), and determining whether planning has accounted for them and either built in hedging strategies or is flexible enough to compensate if they occur.
Although making quantitative estimates of response reliability will prove challenging in many circumstances, there are significant advantages to doing so:
Of course, we would prefer that our preparedness and response systems perform effectively at every incident; however, there is no simple answer to the question "how reliable should they be?" Very reliable systems that are able to perform in the most demanding circumstances cost more than those that can do the same things but less predictably. Investments needed to increase the reliability of response must therefore be measured against ways those funds could be used to achieve different homeland security or other national goals. The first step to making those trade-offs, however, is developing better ways to evaluate the reliability of the preparedness and response systems we have now, and the likely effect of further investments in those systems. Given the absence from policy debate of accurate and objective measures of preparedness and response reliability, the full implications of preparedness investments have not been considered. Framing preparedness policies using concepts such as response reliability has the potential to enrich national debate in this area—moving beyond mere argument about funding levels to a more productive discussion of the trade-offs between investments and the levels of performance that we can reasonably expect from our national preparedness system.