Measuring and Evaluating Local Preparedness for a Chemical or Biological Terrorist Attack

Ronald D. Fricker, Jr., Jerry O. Jacobson, and Lois M. Davis

People in the United States have been terrorized on U.S. soil with targeted violence, on various scales and with varying success, for decades.[1] Yet, the recent September 2001 attacks and the earlier bombing of Oklahoma City's Alfred P. Murrah Federal Building vividly demonstrate something disturbing and new: Some contemporary terrorists, both home-grown and foreign, are not deterred by—indeed may even be attempting to produce—human casualties on a massive scale. These events are part of a pattern of increasing lethality that started in the 1990s.[2] Even more troubling is that, while conventional explosives may continue to be the predominant terrorist weapon, the magnitude of the September 11 attacks coupled with the spread of anthrax-laced letters have led some to conclude that chemical or biological weapons are now more likely than ever to be used.[3]

Although, strictly speaking, terrorism[4] has never been absent from the set of threats facing American citizens, the magnitude and character of recent events have called into question the readiness of the nation's state and local emergency response and health and medical personnel to respond effectively to the next incident, to correctly identify hazards as they occur, and to mitigate damage to persons and property. In particular, response to the anthrax incidents subsequent to September 11 was less than reassuring. While the mode of attack made initial identification and response difficult, authorities also had a hard time organizing, coordinating, and communicating an appropriate response even after anthrax was identified.[5]

This issue paper has two purposes: (1) to suggest some nationally representative measures of local responder preparedness for chemical and biological terrorism as a baseline for the current debate; and (2) to illustrate the limitations of our measures and describe why quantifying preparedness for terrorism, by any measure, is elusive. However, we also note that even perfect measures are not sufficient for managing preparedness improvement. A sound framework that balances investments in preparedness with competing national needs, and that explicitly acquiesces to an acceptable level of risk, is essential and lacking.

The deliberations thus far remain inadequate for the construction of any sound public policy for allocating resources for preparedness, measuring the results, and balancing preparedness needs with other policy objectives. Such deliberations require some measure of preparedness as an input, but the more important question of acceptable risk is a largely philosophical one that requires further debate. Some groundbreaking suggestions have been offered,[6] but more research and discussion are required.

Attempting to Assess Domestic Preparedness Via a Survey

RAND has been collecting data to help address questions of WMD preparedness. From March to September 2001, RAND fielded a survey about domestic terrorism involving WMD to a national cross-section of over one thousand state and local[7] organizations: fire departments, local law enforcement organizations, emergency medical services (EMS), local offices of emergency management (OEMs), local public health departments, hospitals, state EMS offices, state OEMs, and state public health agencies. The survey was designed to elicit the views of organizations that would be involved in the detection of and response to a domestic WMD terrorism incident. All but a few of the surveys were completed and returned prior to September 11, 2001.[8] (See the survey sidebar for additional information.)

Within the survey, respondents—who were either the heads of local response organizations, such as fire or police chiefs, or their designated representatives—were asked both subjective and objective questions about their preparedness for four specific WMD incident scenarios: conventional explosives, and chemical, biological, and radiological weapons. We defined the size of the scenarios so that they were approximately the magnitude of the 1995 Oklahoma City attack. (See the WMD scenarios sidebar for the complete chemical and biological scenarios.) Incidents of the magnitude of the Oklahoma City bombing were chosen because they could not be dismissed as too large to be expected and yet they were large enough to represent major events.[9]

In this issue paper, we focus on two types of WMD incidents: chemical and biological. We further focus on two measures of preparedness: whether the response organization has a plan to address the particular incident and, among those with plans, whether the plan has ever been exercised. We then look across the country and evaluate preparedness in terms of organizations that have plans and have conducted exercises for scenarios of the scope and size we proposed. The assumption implicit in such an approach is that preparing for events similar to those described in the scenarios is equally important and relevant to every jurisdiction and organization in the United States. This is not likely to be true. Yet, in the absence of authoritative guidance about expected terrorist threats, we offer nationally representative estimates (unadjusted for jurisdictional attributes) as a starting point.

Figure 1

Figure 1. Percentage of Each Type of Local Response Organization with a WMD Plan That Addresses Response to a Type of Incident Similar to Those in the WMD Scenario Sidebar

Existence of Plans for WMD as a Preparedness Measure

We begin by focusing on whether an organization has a written plan in place to address each of the chemical and biological scenarios described in the WMD scenarios sidebar.[10] As we previously mentioned, compared with the attacks of September 11, these scenarios are relatively modest, resulting in less then 200 casualties. The existence of a plan is used as one indicator of preparedness, although a plan in and of itself guarantees little about whether an organization can actually execute a successful response. In the ideal case, a plan is evidence that an organization has carefully thought through how it will respond to a particular WMD incident. At the other extreme, the plan may represent nothing more than the rote fulfillment of a bureaucratic mandate. However, it is reasonable to believe that an organization with a plan is at least marginally better prepared than an equivalent organization without a plan.

Figure 2

Figure 2. Percentage of Each Type of State Organization with a WMD Plan That Addresses Response to a Type of Incident Similar to Those in the WMD Scenario Sidebar

Our survey results show that few organizations have a plan in place sufficient to address our moderately sized chemical and biological incidents (Figure 1). In fact, we estimate that fewer than one-third of all local organizations have plans that address either scenario, with the exception of OEMs' and hospitals' plans for chemical WMD incidents.[11] The figure also shows that planning for chemical WMD incidents tends to be more common than planning for biological incidents, which might be expected given the similarity of chemical attacks to other, more common accidental incidents, such as hazardous materials releases.[12] In contrast to local planning, a significantly larger fraction of state organizations have plans (one-half to three-quarters) for these incidents (Figure 2).

Consistent with the types of incidents each organization generally expects to encounter, public health organizations' plans more often address the biological scenario, whereas the plans maintained by OEMs and EMS providers more often address the chemical scenario. This result also suggests that "first-responder" organizations—police, fire, and EMS—and offices of emergency management lag public health departments in planning for bio-terrorist incidents.

Evaluating Plan Quality

To get some insight into the quality of the plans, we asked more-detailed questions about various aspects important for responding to moderately sized WMD incidents. For instance, we asked whether a plan addressed (1) communication procedures with other organizations during the response, (2) procedures for mass decontamination, (3) isolation and quarantine, and (4) coordination with agencies from other jurisdictions. We found that there are far fewer organizations with plans that address these important aspects of responding to a moderately sized chemical or biological incident. For example, we found the following for those with plans:

  • Communication Procedures. Plans generally address communication with other local response organizations. The range for a chemical incident is from 87 percent of local public health agencies' plans to 100 percent of the EMS organizations' plans. For biological incidents, the results are similar except for law enforcement agencies' plans, in which only 59 percent address communications with other organizations.
  • Mass Decontamination. Fewer plans address mass decontamination: Fifty to 65 percent of local fire departments', law enforcement agencies', EMS', and OEMs' plans address mass decontamination for a chemical incident of the size described in the scenario. Even fewer address mass decontamination for the biological scenario: between 35 and 60 percent. Least likely to address mass decontamination are the plans of state EMS and OEM organizations: For either type of scenario, the percentage is between 35 and 45 percent.
  • Isolation and Quarantine. Fifty-six percent of hospitals' plans address access to an isolation/quarantine facility for patients exposed to an infectious biologic agent. Thirty-five percent of local public health plans and 19 percent of state public health plans address this issue.
  • Coordination. Virtually all (more than 90 percent) of state OEMs' plans address coordination with agencies in other jurisdictions for both chemical and biological incidents. The same is true for the majority of local responders: For chemical incidents, the percentages range from 69 percent of local public health organizations' plans to 91 percent of law enforcement plans; for biological incidents, the range is from 53 percent of law enforcement agencies' plans to 78 percent of OEMs' plans.

An Additional Measure of Preparedness: Exercises

Since WMD events are extremely rare, maintaining capabilities and awareness of the contents of written plans through live or tabletop exercises is critical to sustaining readiness over time. However, as with plans, exercises can vary in the capabilities they test and in their content, realism, sophistication, and overall quality; so simply knowing an organization has conducted an exercise is not perfect evidence that an organization is actually prepared to respond to a WMD incident. Nonetheless, as with plans, it is reasonable to assume that organizations that have conducted some sort of exercise are likely to be better prepared than those that have not. Based on the survey results, we find that the percentage of local response organizations that both have a plan and have exercised it within the last two years for either a chemical or biological WMD incident is small (see Figure 3).

Figure 3

Figure 3. Percentage of Each Type of Local Organization That Both Has a WMD Plan and Has Exercised It in the Last Two Years for the Type of Incident Indicated

As the figure shows, plans and exercises are more common for chemical incidents than for biological incidents. However, other than hospitals and offices of emergency management, fewer than 15 percent of any of the other organizations both have a plan and have exercised for a chemical incident in the last two years. For a biological incident, that percentage drops to less than 7 percent for any type of organization.[13]

Furthermore, even for those that conducted exercises, many of the exercises did not address important aspects of responding to a WMD incident. For example, less than half of the biological WMD exercises conducted by fire and law enforcement organizations tested how they would communicate with hospitals and public health organizations. One-quarter of hospitals' and local public health agencies' biological incident exercises did not test how they would communicate with law enforcement, fire, and EMS agencies.

Similarly, the majority of these exercises did not test each organization's ability to coordinate with other agencies outside of the organization's jurisdiction, nor how it would coordinate with its local public health department in the case of a biological incident. In addition, with the exception of law enforcement chemical incident exercises, less than half of the chemical or biological exercises by fire, law enforcement, and EMS agencies tested the capability to decontaminate victims. (Seventy-five percent of law enforcement chemical exercises included decontamination.)

Difficulties in Measuring Preparedness

We have described a possible set of preparedness measures based on two possible WMD scenarios. Preparedness, of course, can be assessed only for a specified event of a given magnitude. A building in a given locale, for instance, may be rated to withstand an earthquake of up to 7.0 on the Richter scale, but not more. The analogy for a terrorist attack is not academic. Personnel may be trained to recognize signs and symptoms of anthrax infection, for example, but not exposure to soman nerve gas. A fire department may be equipped and trained to save some fraction of occupants in a given building struck by an explosion of a given intensity. The implications for measurement and evaluation of preparedness are clear: Assessments must anticipate and prioritize the events and circumstances for which preparedness is desired.

Apart from being specific to a given scenario (or set of related scenarios), the ideal preparedness assessment would be based on objective measures of response performance (e.g., personnel's correct identification of the hazard, correct selection of personal protective equipment, appropriate care in evidence collection at the scene, response time for each step, and timely communication with other required response entities). Short of after-action reports following response to an actual incident, such an exercise is perhaps the most valid proxy for preparedness. However, even though every jurisdiction faces some level of WMD terrorism risk, it is not immediately clear whether such full-blown exercises for WMD terrorism are justified nationally. Absent a credible estimate of the expected cost of terrorism over time, there can be no reasonable upper or lower bound to investments in measurement and evaluation of preparedness.

Furthermore, it is clear that a mail or telephone survey cannot provide an accurate measure of preparedness. Data collected in the RAND survey, such as the existence of plans and the conduct of exercises, imperfectly measure actual preparedness. However, such survey data are useful in that they do provide some objective, representative evidence about the level of ongoing preparedness activities throughout the country. Although the absence of written plans and exercises does not mean an organization would not rise to the occasion and effectively respond to a WMD incident, it does raise serious doubts. It would certainly be reasonable to question whether an organization that does not have plans and has not practiced has done all it can to respond most effectively to an incident in order to mitigate losses.

The survey's chemical and biological scenarios were defined to be as relevant as possible to all jurisdictions across the United States. For example, even regions lacking an office building large enough to hold 200 people, as in the chemical incident scenario, are likely to have events that aggregate 200 people, such as Sunday services at a church or a county fair. Similarly, places that do not have a regional airport, as in the biological incident scenario, probably do have train or bus stations.[14]

However, these scenarios are likely to be more relevant to large urban areas and less relevant to rural areas. Survey results show that local response organizations (particularly large organizations) in areas that have either had a past incident or had a large event that caused the organizations to confront the possibility of a terrorist event are more likely to have plans in place for a chemical or biological WMD event and to have exercised that plan. Arguably, those organizations most likely to face the threats posed in the scenarios are more prepared. Thus, it is possible to conjecture from the survey results that local preparedness may already be rationally apportioned to those areas that are more likely to be threatened.

Nevertheless, our survey results are less optimistic than the results from a National League of Cities (NLC) questionnaire that was faxed to cities immediately after September 11. As reported in The Washington Post,[15] NLC received 456 returned questionnaires out of 1,800 faxed to cities and towns. Of the cities and towns that responded to the NLC survey, 55 percent indicated they had "terrorism response blueprints" in place. Our results show that, even if the NLC survey results are representative of all cities and towns in the United States,[16] a much smaller fraction of the actual response organizations within the cities and towns have emergency response plans that address moderately sized chemical and biological WMD incidents. Furthermore, our results show that only a fraction of organizations with plans have exercised them and that those exercises often do not test important aspects of responding to a WMD event.

In addition, beyond these potential operational difficulties are foundational matters of domestic preparedness strategy for which debate has hardly begun: What does "preparedness" for chemical and biological weapons, and for weapons of mass destruction (WMD)[17] more generally, mean? How much preparation do we need? To put it more bluntly, how much should be invested in risk mitigation measures for an as yet largely undefined terrorist threat? Given the unpredictability of terrorist acts and the political and societal interests involved, these are hard questions to address.

While we have pointed out some key difficulties in measuring preparedness, even if a perfect metric existed that could predict an organization's ability to respond to any type of WMD incident, it would still not be obvious how to apportion additional resources and fiscal assistance. Not only is resource apportionment among domestic response organizations difficult and important, but it is also difficult to assess how much that same fiscal investment could save if spent elsewhere: on military action routing terrorists oversees, for instance, or on foreign policy, freezing terrorists' assets, or even on preparing for more frequent natural hazards or other national needs entirely unrelated to terrorism.

Furthermore, because of the lack of authoritative threat and outcome assessments, preparedness or lack thereof in any particular jurisdiction is largely a matter of subjective opinion. Without comprehensive threat assessments, it is exceptionally difficult to define how much preparation is enough and hence specify what "appropriate" preparedness is for any jurisdiction.

This is not a new point. It was examined by Richard Falkenrath in a discussion paper in December 2000,[18] debated by the Gilmore Commission, and subsequently published in International Security.[19] As Falkenrath suggested,

This lack of broad but measurable objectives [for domestic WMD preparedness] is unsustainable. It deprives policymakers of the information they need to make rational resource allocations and renders program managers unable to measure genuine progress. It also suggests endlessly escalating program expenditures, because a process whose only goal is to improve from current standing has no logical conclusion.

In conclusion, using the RAND survey data, we have provided nationally representative estimates of the existence of written plans and exercises for moderately sized domestic WMD chemical and biological attacks. Overall, the number of response agencies with plans that they had exercised within the past two years is small relative to the number of organizations surveyed. However, we also find that large organizations are more likely to be prepared than small organizations, and urban areas are more likely to be prepared than rural areas, which may reflect a reasonable distribution of preparedness since organizations and regions likely face differing threats.

Our inability to reach more definitive conclusions is indicative of a larger problem policymakers face: Without a well-defined, consistently applied national methodology for providing threat assessments and assessing preparedness, it will be difficult, if not impossible, to judge how prepared the country is to meet a specific WMD terrorism threat inside its borders, to manage rationally the apportionment of resources to improve preparedness, and to judge whether preparedness goals have been met. The challenge for policymakers is to formalize and systematize threat assessments and preparedness measures in order to provide a rational basis for future WMD preparedness policy decisions.

Notes

[1] Terrorism has been perpetrated by many types of groups, including right-wing militia organizations, the Ku Klux Klan, activists such as Theodore Kaczinski, ecoterrorists, and extremist antiabortion and animal rights groups.

[2] See Bruce Hoffman, "Terrorism Trends and Prospects," in Ian O. Lesser, Bruce Hoffman, John Arquilla, David F. Ronfeldt, Michele Zanini, and Brian Michael Jenkins, Countering the New Terrorism, Santa Monica, Calif.: RAND, MR-989-AF, Ch. 2, 1999.

[3] For example, The Washington Post reported, "Bush administration officials said yesterday there will likely be more terrorist strikes in the United States, possibly including chemical and biological warfare" (D. Milbank, "More Terrorism Likely, U.S. Warns," The Washington Post, October 1, 2001, p. A01). Similarly, The Washington Post later reported that "Top health officials warned that the anthrax attacks are still escalating" (M. Powell and C. Connolly, "Experts Warn Bioterrorism Could Expand: N.Y. Hospital Worker Dies of Anthrax," The Washington Post, November 1, 2001).

[4] We use the term "terrorism" here to mean the unlawful use, or threat of use, of force or violence for political purposes, where such force or violence is intended to create an atmosphere of fear in order to coerce others into actions they otherwise would not undertake or into refraining from actions that they desire to take. This definition captures most of what is central to the various formal definitions. (For a short compilation of definitions from academia and government, see www.terrorism.com/terrorism/def.shtml)

[5] See, for example, Sue Ellen Christian, "Anthrax Scare: Health System's Weakness Exposed," Chicago Tribune, November 18, 2001; Jeff Nesmith, "U.S. Health Officials Rapped on Anthrax," The Atlanta Journal and Constitution, November 30, 2001; "Are We Ready? Public Health System Needs Upgrading," The Dallas Morning News, October 29, 2001; and Robert Davis, "Local Health Agencies Unprepared for Bioterrorism," USA TODAY, October 29, 2001.

[6] See, for example, Richard Falkenrath, "The Problems of Preparedness: U.S. Readiness for a Domestic Terrorist Attack," International Security, Vol. 25, No. 4, 2001, pp. 147­186. Also see the second and third annual reports of the Advisory Panel to Assess Domestic Response Capabilities for Terrorism Involving Weapons of Mass Destruction. The panel has submitted three annual reports to the Congress and the President of the United States as of this writing, one each in December of 1999, 2000, and 2001. They can be accessed online at www.rand.org/nsrd/terrpanel.

[7] Local organizations, depending on the type of organization, may serve a town, city, or county.

[8] The results in this issue paper are based only on the surveys received prior to September 11, 2001.

[9] Prior to September 11, the particular chemical and biological scenarios we posed were considered relatively large, because they reflected the magnitude of the largest incident at that time. Post-September 11, these scenarios now appear to be of a moderate size. This is, of course, a judgment that is relative to the respondent's frame of reference. Those respondents from more rural regions are likely to view the scenarios as relatively large, while those from large cities are likely to view the scenarios as moderate.

[10] Respondents were presented with each scenario and asked to decide for themselves whether any of their organization's written plans "addressed" response to the scenario.

[11] All numerical estimates reported in this issue paper, except those for local EMS, have been statistically adjusted so that they are representative of all of the organizations of a particular type throughout the entire United States. Local EMS surveys were sent to a convenience sample and thus cannot be generalized beyond the actual survey sample.

[12] While the observed differences between the chemical and biological scenarios could also be due to differences in the perceived magnitude of the specific scenarios presented in the survey and/or sampling error, less emphasis on biological WMD is consistent with anecdotal evidence and expert opinion. See, for example, Bioterrorism: Federal Research and Preparedness Activities, Washington, D.C.: General Accounting Office, GAO-01-915, September 2001, or ABCNews.com at http://abcnews.go.com/sections/us/DailyNews/WTC_chemicalbiologicalqa.html: "Most experts say the United States is not adequately prepared [for a biological attack, but] most experts believe the United States is adequately prepared for a chemical attack" (accessed December 27, 2001).

[13] Furthermore, this picture does not change substantially when looking at whether the plans were ever exercised. For a chemical incident, the percentages for OEMs and hospitals increase to 38 and 42 percent, respectively, and the other local response groups are all less than 20 percent. For a biological incident, the changes are even less striking. Public health and hospitals increase to 11 and 16 percent, respectively, and all of the other local response organizations remain less than 7 percent.

[14] Furthermore, in the case of agricultural areas, this type of threat may not be restricted to human beings, but also may occur as an agro- terrorism incident in places where livestock are aggregated or transported.

[15] "Many U.S. Cities, Towns Not Ready for Terrorism," The Washington Post, September 23, 2001, p. 44.

[16] There is reason to doubt that the NLC survey results are representative, since those cities and towns that are prepared are more likely to respond to such a survey. Given that the NLC survey response rate was only 25 percent, it is likely that the actual fraction of cities and towns with "terrorism response blueprints" is smaller—perhaps significantly smaller—than 55 percent.

[17] We define WMD as any devices capable of producing large-scale physical destruction, widespread disruption, and/or mass casualties. As such, WMD may be chemical, biological, radiological, or nuclear devices or weapons. They could also entail the use of conventional explosive devices or involve, for example, an attack on an industrial facility resulting in the release of a toxic substance into the surrounding community.

[18] For a more detailed discussion of this point, see Richard A. Falkenrath, The Problems of Preparedness: Challenges Facing the U.S. Domestic Preparedness Program, Cambridge, Mass.: John F. Kennedy School of Government, Harvard University, BCSIA Discussion Paper 2000-28, December 2000.

[19] Richard A. Falkenrath, "The Problems of Preparedness: U.S. Readiness for a Domestic Terrorist Attack," International Security, Vol. 25, No. 4, 2001, pp. 147­186.

IP-217-OSD (2002)