- What are the sources of potential bias in the results of the 2014 RAND Military Workplace Study?
- What effect do these potential biases have on the study's results?
In early 2014, the Department of Defense Sexual Assault Prevention and Response Office asked the RAND National Defense Research Institute to conduct an independent assessment of the rates of sexual assault, sexual harassment, and gender discrimination in the military — an assessment last conducted in 2012 by the Department of Defense using the Workplace and Gender Relations Survey of Active Duty Members. The resulting RAND Military Workplace Study invited close to 560,000 U.S. service members to participate in a survey fielded in August and September of 2014. This volume presents the results of methodological investigations into sources of potential bias in the survey estimates for active- and reserve-component service members. It includes evaluations of follow-up studies of survey nonrespondents and the efficacy of sampling weights to correct nonresponse bias, an assessment of total survey error using an administrative records benchmark, estimates of potential under- and overcounting of service members exposed to sexual assault, comparisons of events identified by prior survey forms and the RAND forms, analysis of survey non-consent and breakoff, and evaluation of service member tolerance of the RAND forms. In the final chapter, the report draws conclusions and recommendations for future administrations of sexual assault and harassment surveys in the military.
We found no conclusive evidence of substantial bias or error in previously reported RMWS estimates
- There was, however, a general pattern that suggests the primary RMWS estimates of sexual assault, sexual harassment, and gender discrimination are more likely to be underestimates than overestimates of true population values.
We found little evidence that the study overcounted rates of sexual assault, sexual harassment, and gender discrimination
- For example, although we concluded that a small number of pre-service sexual assaults may be captured in our estimates, this number is almost certainly lower than the larger number of assaults that go uncounted because we excluded members with fewer than six months of service and those who left the military shortly before the survey fielded.
A substantial portion of any underestimation of sexual assault rates in the military is attributable to our use of a sample frame that excludes some high-risk service members
- The strongest evidence for bias comes from the fact that the survey sample frame clearly excluded some individuals who served in the military in the past year and who may have experienced a sexual assault. This source of bias may explain a substantial proportion of the total survey error identified by our comparison of survey-estimated counts of reported sexual assault to official reports of sexual assault.
We report evidence that the methods we used to weight survey respondents produced less-biased estimates than the conventional approach to weighting military surveys
- In particular, the analysis of nonresponses weights found that the conventional sample weighting methods resulted in the underrepresentation of a number of groups of service members who have a high risk for sexual assault and harassment.
- Future surveys of sexual assault and sexual harassment should use the RMWS measurement approach, or comparable survey questions that use behaviorally and anatomically specific language to clearly define criminal sexual assault and violations of equal opportunity law and policy.
- Omitting service members who recently separated from the military could lead to significant bias in estimates of past-year sexual assaults, sexual harassment, and gender discrimination. As a result, we recommend including past-year separations in the sample frame of future surveys, or developing analytic approaches for estimating the number of crimes and violations those who separated experienced in the past year.
- The Defense Manpower Data Center (DMDC) should design future surveys to include sufficient numbers of men in the sample to ensure ongoing assessment of the nature of sexual assaults against them. In practice, this means large sample surveys that may not oversample women at rates as great as in the RMWS or previous WGRA studies. This can be done without reducing the precision of women's estimates below those of men.
- DMDC should build on approaches developed for the RMWS to include a wider set of factors in future nonresponse weighting models than has previously been possible for military surveys.
- The Office of the Secretary of Defense (OSD), the Defense Information Systems Agency, and the services should collaborate to improve the coverage and reliability of email contact information in the personnel systems used for survey recruitment. Also, DMDC should investigate additional modes of recruitment (phone or text message) that improve outreach to members who do not routinely use email as part of their military duties.
- In future surveys on sexual assault, DMDC should continue to compare survey estimates with actual numbers of filed Victim Reporting Preference forms as a measure of nonresponse bias and total survey error more generally.
- OSD should conduct the survey no more frequently than once every two to four years.
Table of Contents
Follow-Up Studies of Survey Nonrespondents
The Efficacy of Sampling Weights for Correcting Nonresponse Bias
Investigation of Total Survey Error Using Official Records of Reported Sexual Assaults
Performance of the Sexual Assault Survey Module
Undercounting and Overcounting of Service Members Exposed to Sexual Assault
Performance of the Sexual Harassment and Gender Discrimination Module
Comparison of Events Identified by the Prior Form and RAND Forms
Analysis of Survey Nonconsent and Breakoff
Service Member Tolerance of the RAND Form
Conclusions and Recommendations for Future Administrations of the WGRA
Phone Survey Script
Mail Survey (Male and Female Respondent Versions)
Supplementary Tables for Chapter Three
Supplementary Tables for Chapter Seven
This research was conducted within the Forces and Resources Policy Center of the RAND National Defense Research Institute, a federally funded research and development center sponsored by the Office of the Secretary of Defense, the Joint Staff, the Unified Combatant Commands, the Navy, the Marine Corps, the defense agencies, and the defense Intelligence Community.
This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.