Sep 13, 2022
The Women's Reproductive Health Survey (WRHS) represents the first survey across the U.S. Department of Defense (DoD) of only women since 1998. In part, the WRHS fulfills requirements of the 2016 and 2017 National Defense Authorization Acts (NDAAs). These acts include several requirements on the provision of health care to service women, as well as for a survey of service members' experiences with family planning services and counseling. This brief reviews WRHS development and administration, the population and sample, survey weights, imputation of missing data, analysis approach, and limitations. Readers seeking more information may consult the full report or a summary of its findings.
The only statutory content requirements for survey content were those from the 2016 and 2017 NDAAs. Given these minimal requirements, the RAND Corporation research team reviewed other existing surveys related to women's health generally and reproductive health specifically. From these, team members devised a list of survey topics and items that they shared with the sponsor and an advisory group of roughly a dozen military and civilian subject-matter experts. Institutional review boards for the protection of human subjects reviewed the final survey and related protocols.
RAND partnered with Westat, which implemented the confidential web-based survey. The survey opened in August 2020 and closed in November 2020. The initial contact with sample members included a mailed invitation from RAND and Westat, a letter from the Assistant Deputy Secretary of Defense for Health Affairs, and a set of frequently asked questions. All sample members—active-duty service women (ADSW)—with a valid .mil DoD email address also received an introductory email. After that initial contact, ADSW who had not completed the survey or who had not opted out of future communication received up to five additional email reminders or one additional postal mail reminder. The Defense Manpower Data Center provided all contact information. ADSW could also check their eligibility to take the survey on a Westat website. The survey did not require a Common Access Card (CAC), although all eligible sample members did receive a unique code to enter the survey. All confidentiality procedures were outlined in the survey consent and start screens.
The study population consisted of ADSW in the Air Force, Army, Marine Corps, Navy, and Coast Guard, excluding trainees and ADSW above flag rank. The total sampling frame was 249,128 ADSW. A random sample of DoD ADSW was selected, stratified by service branch and pay grade, and sampled proportional to size from the strata, with a total sample size of 125,000. For the Coast Guard, a census of all 6,113 eligible ADSW was used. The final analytic sample consisted of 23,950 completed surveys in which the respondent completed at least 50 percent of the items up to the end of the survey section on contraceptive access. The overall response rate was 17.8 percent, varying widely both by service and pay grade (see Figure 1).
The researchers weighted the data to account both for survey design and differential nonresponse by subgroups. A service member's design weight is the inverse of her probability of being selected for sampling. The research team computed nonresponse weights as inverse probability weights, using a statistical model to estimate the probability of an individual being a respondent. Analytic weights, used in all analyses, are the product of the design and nonresponse weights. Use of the weights results in a representative sample on many important sociodemographic and military characteristics (e.g., age, race/ethnicity, marital status, education level, number of dependents, service branch, pay grade, years of service, occupation code, recent deployment, and Armed Forces Qualification Test scores).
The nonresponse weighting accounts for differences between those who responded to the survey and those who did not but does not account for survey drop-off (stopping the survey before reaching the end) or intermittent missing data (skipping an item). To account for both types of missing data, the research team used a sequential imputation model with predictive mean matching for all variable types. Prior to implementing this, the researchers performed a logical imputation that accounted for skip logic and known logical relationships among the items.
To test for differences in key outcomes across different characteristics—service branch, pay grade, race/ethnicity, age group, marital status, and sexual orientation—researchers used a two-stage procedure. First, they conducted a Rao-Scott chi-square test for overall differences across categories within a subgroup. Second, if the overall test was statistically significant, the researchers then used the Wald test to examine differences across each pairwise comparison of categories within the subgroup. To adjust p-values for multiple comparisons, the researchers used the Tukey-Kramer method, which accounts for multiple tests associated with all possible pairwise comparisons of categories within a subgroup. The research team suppressed both point estimates and confidence intervals when fewer than 15 ADSW were in an analytic cell. The team suppressed confidence intervals when the half width of a confidence interval was greater than the point estimate itself.
There are several limitations to this research. First, WRHS response rates were low, although higher than in many recent DoD-wide surveys. Second, for some groups (e.g., recently deployed Marine Corps ADSW), sample sizes were small. Third, the survey language did not specify whether care provided within the Military Health System refers only to direct care at military treatment facilities or also includes care elsewhere through TRICARE. It also did not explicitly include Coast Guard clinics as part of Military Health System care. Fourth, the survey was fielded during the COVID-19 pandemic, although the impact of the pandemic on the study is not necessarily obvious. The pandemic might have limited access to DoD email for remote workers or negatively influenced perceptions of available care, but it also might have led to greater schedule flexibility, affording time for respondents to complete the survey.