The National Aeronautics and Space Administration (NASA) confronts a variety of organizational-level risks within its programs that could influence the success of its missions or programs. This report provides NASA's Office of Strategy and Plans with a risk assessment methodology that integrates risk factors tailored to NASA's management, operations, and acquisition structures, allowing decisionmakers to compare risks within a common framework.

Developing a Risk Assessment Methodology for the National Aeronautics and Space Administration

Daniel M. Gerstein, James G. Kallimani, Lauren A. Mayer, Lelia Meshkat, Jan Osburg, Paul K. Davis, Blake Cignarella, Clifford A. Grammich

ResearchPublished Jun 29, 2016

Cover: Developing a Risk Assessment Methodology for the National Aeronautics and Space Administration
Download PDF
Additional Downloads

Includes Appendix B and C

The National Aeronautics and Space Administration (NASA) confronts a variety of organizational-level risks within its programs that could influence the success of its missions or programs. Comparing, contrasting, and mitigating these risks require developing a common lens through which to view them. Such an evaluation can increase overall understanding of the risks associated with NASA-level decisions. This report provides NASA's Office of Strategy and Plans with a risk assessment methodology that integrates risk factors and risk management approaches tailored to NASA's management, operations, and acquisition structures. While NASA has deep experience in conducting risk assessments on highly technical issues for individual programs and projects, it has not developed an overarching methodology for thinking about overall NASA-level risk associated with its decisions. For this effort, a methodology was developed for normalizing and comparing supply chain, external dependency, cost and schedule, human capital, organizational and managerial, political, and technical risks. The goal ultimately was to develop a single, overarching, risk-informed decision support methodology for looking at disparate risks through a common lens. The methodology presented is for a non-expert audience so that any practitioner or decisionmaker with any level of training can use it.

Key Takeaways

Seven Key Risk Factors Examined

  • The risk factors considered were: supply chains, dependency on external sources, acquisition costs and schedules, workforce issues, organizational and managerial considerations (e.g., the issues arising from NASA's distributed management between mission directorates and research centers), domestic and international political situations (e.g., partnership instabilities, congressional equities), and technical issues including information security.

Structured Methodology Provides Benefits Beyond Final Calculations

  • Considering not only risks but also mitigations at an early stage in the analysis can prompt a structured thinking process that allows for clearer and more-thoughtful decisions.
  • The use of a normalization process allows comparison of disparate risk issues that contribute to important decisions. Without such normalization, decisionmakers would be presented with a number of risk factors using different scales (and likely different grading and weighting) with little ability to understand comparisons within or among them or the overall risk burden associated with a decision.
  • The result of such an analysis is a graphical representation of normalized relative risks designed to allow for understanding the NASA organizational-level risk associated with its decisions.

Case Studies Validate Suggested Methodology

  • Each discrete risk assessment requires that the methodology be changed to reflect the particulars of the issue under consideration.
  • Two decisions — one past and one future — were considered as a way to test and partially validate the methodology. The previous decision considered was the cancellation of the Space Shuttle program and commercialization of the transportation to the International Space Station, while the future decision focused on Cislunar habitation.

Recommendations

  • Each discrete assessment requires that the methodology be changed to reflect the particulars of the issue under consideration.
  • Extensive resources may be required to develop the inputs for this methodology but once inputs are in hand, the significant remaining task is to elicit weights from decisionmakers. Data for each risk factor may be obtained from different experts in each area. Therefore, while resource-intensive, much of the work may occur concurrently.
  • Separating the experts from the decisionmakers is imperative for assuring the methodology produces unbiased results. Experts are essential in developing the risk factor component charts; decisionmakers are essential in developing weights. Decisionmakers should approach the methodology without being encumbered by the discussions associated with completing the chart and with the opportunity to synthesize the material and think at a higher level of abstraction about the results. That is, there should be an independent role for the decisionmaker, reflecting a broader pattern of thinking on a risk-informed decision issue.
  • The steps developed in the methodology provide a structured way to consider a risk-informed decision. While we were able to conduct abbreviated case studies validating it, conducting a more robust analysis for a future NASA decision is important. Given the work we have begun on it, applying this model to a more expanded analysis of the pending Cislunar habitation decision may be most appropriate. Such work would provide further validation of the methodology and additional insights into the risk (and options) associated with this decision.

Topics

Document Details

Citation

Chicago Manual of Style

Gerstein, Daniel M., James G. Kallimani, Lauren A. Mayer, Lelia Meshkat, Jan Osburg, Paul K. Davis, Blake Cignarella, and Clifford A. Grammich, Developing a Risk Assessment Methodology for the National Aeronautics and Space Administration. Santa Monica, CA: RAND Corporation, 2016. https://www.rand.org/pubs/research_reports/RR1537.html.
BibTeX RIS

The research reported here was prepared for the National Aeronautics and Space Administration and conducted by the Science, Technology and Policy Program within RAND Justice, Infrastructure, and Environment.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.