The U.S. Coast Guard needs to measure its operational-level performance effectively. To assist this effort, the authors created logic models detailing elements of the Coast Guard's 11 missions. After identifying and assessing the validity, reliability, and feasibility of relevant existing metrics, the authors go on to propose potential metrics that could improve on or complement existing metrics, as well as a framework for applying the metrics.
- What aspects of operational performance should the Coast Guard aim to measure?
- How well does the Coast Guard measure operational performance?
- How could the Coast Guard better measure operational performance?
The U.S. Coast Guard needs to measure its operational-level performance effectively to make informed decisions about resource allocation. To assist the Coast Guard in this effort, the authors worked with official documentation and subject-matter experts to develop logic models describing each of the 11 statutory missions of the Coast Guard and using the descriptions to ascertain what aspects of these missions should be measured. The authors examined existing metrics in the light of these logic models, evaluating the metrics in terms of their validity (how well they measured elements of the logic models), reliability (how consistently measurements can be made), and feasibility (how readily measurements can be made). They also analyzed the extent to which existing metrics measure elements of the logic models. They then developed and evaluated sets of potential metrics that could improve on or complement the existing metrics, together with a framework for applying metrics in decisionmaking. This report describes the logic models, existing metrics, and potential metrics, including their relationships with one another and their derivation from Coast Guard sources. Note that, as of this writing, the Coast Guard is considering these findings and has not adopted them as doctrine.
This Research Produced a Series of Logic Models Describing Coast Guard Missions
- Using Coast Guard documentation and discussions with subject-matter experts, the authors developed logic models that describe each of the service's 11 statutory missions in detail. These logic models include the outcomes, accomplishments, and activities (collectively termed "elements") associated with each mission, as well as relationships among them.
- The elements of these logic models suggest suitable areas for performance measurement.
Existing Metrics Do Not Capture All the Elements of These Models
- Existing metrics capture only 29 percent of the elements of the logic models.
- Of the existing metrics, 48 percent have high validity, 74 percent have high reliability, and 82 percent have high feasibility; only 34 percent rate highly on all three criteria.
The Authors Developed a Menu of Potential Additional Performance Measures Addressing All Elements of the Models
- The potential measures either cover unaddressed elements or offer higher levels of validity, reliability, and/or feasibility than existing metrics.
- These measures are grouped by mission and aligned with specific elements of the logic models.
- The Coast Guard can select the potential measures that address its most critical operational issues.
The Authors Developed a Framework for Applying Metrics to Make Decisions
- By using a structured framework, the Coast Guard can use metrics more effectively to aid operational-level decisions.
- The logic models offer a means for the Coast Guard to assess what it needs to measure with respect to each mission.
- The service can select from among existing and proposed metrics to find those that are most suitable for addressing operational questions and making decisions.
- The metrics will need to be evaluated to determine appropriate values for individual metrics and relationships among metrics.
- The Coast Guard can also coordinate with other components of the Department of Homeland Security, as well as with the Department of Defense, National Oceanic and Atmospheric Administration, Environmental Protection Agency, Drug Enforcement Agency, and other agencies, to align operational metrics in a way that facilitates communication and decisionmaking.
- The Coast Guard can use a structured framework to facilitate the use of metrics in making operational-level decisions.
Table of Contents
Discussion of Overall Findings
Using Metrics to Support Decisionmaking
Search and Rescue
Ports, Waterways, and Coastal Security
Aids to Navigation and Waterways Management
Marine Environmental Protection
Living Marine Resources
Other Law Enforcement