Sexual Assault and Sexual Harassment in the U.S. Military: Volume 4. Investigations of Potential Bias in Estimates from the 2014 RAND Military Workplace Study
Mar 16, 2016
Photo by Spc. Jacob Kohrs/U.S. Army
In early 2014, the U.S. Department of Defense (DoD) asked the RAND National Defense Research Institute to conduct an independent assessment of sexual assault, sexual harassment, and gender discrimination in the military. The RAND Military Workplace Study (RMWS) included one of the largest surveys of its kind: Almost 560,000 active- and reserve-component service members were invited to participate, and more than 170,000 completed the survey.
The RMWS took a new approach to counting service members who experienced sexual assault, sexual harassment, and gender discrimination, providing DoD with unprecedented detail on the frequency, nature, and context of these experiences among men and women in each branch of service. DoD requested this redesign to better align the survey with legal definitions of sexual assault and sexual harassment.
Because such a large number of service members participated in the survey, the RMWS estimates were very precise, but precision alone does not guarantee that the study correctly counted the events it was designed to assess. Therefore, the RAND team conducted a series of methodological substudies to investigate whether the RMWS survey estimates suffered from other sources of error, including: misclassification of sexual assault and harassment, the exclusion of some types of service members from the study sample, and differences between survey respondents and nonrespondents in their risk of sexual assault and harassment. These investigations included both additional statistical analyses of the previously published RMWS study data and analyses of new data collected in follow-up surveys of service members and from administrative records.
These investigations did not identify substantial bias or error in the previously reported RMWS estimates. However, the evidence suggests that the RMWS estimates were more likely to slightly underestimate than to overestimate the true rate of sexual assault in the military. The exact magnitude of any bias appears to be small and cannot be estimated with precision.
These investigations did not identify substantial bias or error in the previously reported RMWS estimates.
If service members who chose not to participate in the survey had different sexual assault or sexual harassment experiences from those of the weighted sample of respondents, this would result in biased survey estimates. To investigate how nonrespondents differed from respondents, we conducted follow-up surveys of three samples of service members who did not participate in the original RMWS survey: (1) 8,060 randomly selected members who were recruited by phone for a phone interview; (2) 8,083 randomly selected members who were recruited by mail for a self-administered paper survey; and (3) 252,741 who did not respond in time to be included in the estimates but were given additional time to complete the original RMWS survey online. These follow-up surveys did not identify a consistent pattern of bias in the estimated prevalence of sexual assault, sexual harassment, or gender discrimination. Specifically, both the phone and web follow-up surveys revealed possible nonresponse bias in the RMWS estimates, but in different directions: The phone survey suggested that the RMWS overestimated the prevalence of nonrespondents’ experiences, while the web-based follow-up suggested underestimation by approximately the same magnitude. The follow-up mailed survey was stronger methodologically but did not identify evidence of significant nonresponse bias for sexual assault or harassment. It did suggest, however, that the RMWS estimates may have slightly undercounted the rate of gender discrimination.
Service members with characteristics associated with a higher risk of sexual assault had the lowest likelihood of responding to the survey. To address this potential source of bias, we developed new and powerful nonresponse weighting methods that built on those used in the earlier DoD surveys and by the broader survey research community. In a series of statistical analyses, we found that the new weights eliminated more sources of potential bias from the RMWS estimates than the weighting procedures used in the previous DoD studies.
New weighting procedures eliminated more potential sources of bias compared with previous studies.
The RMWS sampling plan, survey questionnaire, and analysis plan all required judgments that affected how we counted incidents and the prevalence of sexual assault. We examined the effects of the following decisions on the RMWS estimate that 20,300 service members experienced a sexual assault in the year prior to the survey:
The RAND survey form and the prior DoD survey form used different sets of questions to assess sexual assaults that occurred in the prior year. Both sets of questions were administered in 2014 so that we could compare the new instrument with the prior instrument. These analyses showed that because of how questions were framed on the earlier form, many respondents reported incidents that occurred more than one year ago, which meant DoD’s method of assessing unwanted sexual contact likely overestimated its prevalence by as much as 20 percent. The DoD assessment method also identified only about half as many service members who experienced penetrative assaults as the RAND form (4,200 versus 7,800). This effect was particularly acute for male service members (1,200 versus 3,700), likely because the RAND form identified more sexual assaults that occurred in the context of hazing or that were not perceived as sexual by the service member. The RAND form, which did not require respondents to know the definition of sexual harassment and correctly apply it to their experiences, identified far more cases of sexual harassment than the form previously used by DoD.
By far, the largest source of potential bias was the exclusion of service members who had separated from the military before the survey sample was drawn.
Fewer than 4 percent of those who began the RMWS survey stopped answering questions, or “broke off,” before reaching the sexual assault module. In comparison, 6.5 percent of those who received the prior DoD form as part of the 2014 RAND study and 13.9 percent of participants in the DoD-administered 2012 survey broke off before the critical questions. Although the behaviorally and anatomically specific language in the RAND form’s sexual assault module offended some service members, this did not lead to a higher level of breakoffs in that section relative to other portions of the survey. The RAND form did lead to more complaints than the prior form, with those at the lowest risk for sexual assault (men, officers, and those in more-senior pay grades) tending to object to the survey language at the highest rates. We received three complaints from sexual assault victims that the survey language triggered distressing memories; all said they discontinued their survey participation. In comparison, 12,210 service members classified as having experienced a sexual assault completed the survey. As such, we concur with recent findings from the National Research Council that the benefits of using behaviorally specific language to more accurately assess sexual assault experiences likely outweigh the distress associated with exposing survey respondents to language they find offensive or disturbing.