Establishing a Research and Evaluation Capability for the Joint Medical Education and Training Campus
Jun 9, 2011
The Base Realignment and Closure Commission recommended the establishment of a joint medical education and training campus (METC) to foster interchangeability and interoperability among medical personnel and units across the services and to gain cost and efficiency benefits. Pursuant to this recommendation, such a campus has been established at Fort Sam Houston, Texas. RAND Health and the RAND National Defense Research Institute were asked to provide technical and research assistance in several areas relating to the implementation of METC, including examining the need for and feasibility of establishing a research and evaluation capability within METC. Such a capability would typically be housed in an office of institutional research (OIR).
METC is a joint military training and education complex. It will consolidate most of the medical enlisted training currently conducted at several military installations. Once fully operational, METC will train more than 100 enlisted medical specialties and will be one of the world's largest medical education and training institutions, with an annual throughput of more than 24,500 students, an average daily student load of more than 8,000, and a total of 1,400 faculty and staff members. Its stated goals are to capture best practices, achieve efficiencies in training, and transform itself into a high-performing "learning" organization.
To answer this question, RAND researchers took two approaches. One was to consider it in light of the organization's goal of becoming a high-performing learning organization. The other was to assess the need in light of accreditation requirements.
High-performing organizations typically adopt models of organizational improvement to guide their strategic efforts and focus on achieving results and outcomes. Most high-performing organizations use an organizational improvement model or methodology, such as those espoused by the Malcolm Baldrige Quality National Award program, Total Quality Management, Lean Production, or Six Sigma. All these models emphasize measurement and analysis of organizational performance and the use of those results for organizational improvement. Thus, to support its goal of becoming a high-performing organization, METC will need to develop and sustain the capability to collect, organize, analyze, and use data on a variety of processes and outcomes to support innovation and performance excellence. In addition, it will need to review these indicators and data and analysis systems periodically and adapt to new or changing environments and stakeholder needs. An OIR would provide METC with such a capability.
Accreditation is an important indicator that an institution delivers a legitimate education that meets standards of quality. Individual programs within an institution may also be accredited, which indicates that an educational program has met specific quality standards. Many of METC's programs require program accreditation. Furthermore, among METC's stated goals is formal accreditation as a degree-granting institution of higher education.
Accreditation bodies increasingly require programs and institutions to develop and implement quality-improvement plans and learning objectives and to provide credible evidence of the value added to student learning and subsequent workforce outcomes. Three accreditation bodies examined in the study specify a variety of quality indicators that may be used for assessment and evaluation of occupational education programs. These indicators include such measures as graduation or completion rates, employment or placement rates, pass rates on professional licensure exams, employer satisfaction, participant satisfaction, and assessment of occupational skills and knowledge. Notably, several of these indicators (licensure exam pass rate, employer satisfaction, placement rate) require follow-up with program graduates and supervisors. Standards related to governance structure, program length, and faculty credentials may pose substantive challenges for METC. Regardless, should METC seek accreditation in the future, it will need a research and evaluation capability to meet the accreditation requirement for institutional improvement plans, embedded assessment, and tracking of a variety of indicators. Thus, both of METC's primary goals argue for an OIR.
To gain a sense of the type of research and evaluation that an OIR might conduct, the researchers examined organizations that have missions similar to that of METC to draw out relevant lessons. Similar organizations include corporate universities that are separate entities within companies; METC's counterpart in the United Kingdom, the Defence Medical Education and Training Agency (DMETA); and federal agencies with a focus on training and development, such as the Veterans Health Administration (VHA).
The results show that although corporate universities differ in scope and function, measurement and evaluation of program effectiveness are always key components, and corporate training leaders devote significant resources and attention to evaluation. They also build evaluation into training programs early by devoting considerable attention to evaluation issues in the program development and planning phase. In addition, best-practice organizations focus on the customer in their evaluation efforts.
For its part, DMETA emphasizes continuous evaluation throughout the training and development cycle. It also coordinates higher levels of evaluation that are the responsibility of the UK's individual military services. Additionally, it collects data and reports annually on performance indicators that are part of the Defence Balanced Scorecard.
The VHA has spent considerable time and effort transforming itself into a high-performing learning organization. It has strongly embraced the need for continuous assessment, feedback, and redesign for all VHA training and development programs and has invested considerable resources in evaluation, performance measurement, and metrics for organizational improvement.
There is a clear need for a research and evaluation capability within METC, and it should be established. METC would benefit from the following recommendations in establishing its OIR. The office should be positioned so that it reports directly to senior leaders, and its director should be part of the senior management team. METC should consider building a centralized data warehouse to track students and indicators of their learning and progress; collecting and reporting data for its balanced scorecard and help translate those results so they can be used for organizational improvement; collecting, analyzing, and reporting basic data on the institution that might be needed for external reporting; and designing and evaluating training programs.