- How can evaluations of complex interventions, such as services for older people in the UK, be improved?
Transforming health and care for older people is complex and demanding. Evaluating such efforts requires a range of approaches and ideas, and a reflection on ways to improve how evaluations are commissioned, completed and used in a changing policy landscape. This report by researchers at RAND Europe summarises three workshops undertaken with evaluators, commissioners of evaluations and services, and those delivering services that are evaluated with the explicit aim of addressing these questions.
- Evaluations were too often not focused on major challenges and consequently findings were 'unsurprising' or even trivial.
- Evaluation findings were often not available when decisions had to be made.
Replication and lack of cumulative building of knowledge
- Evaluations were often designed as stand-alone pieces rather than building on previous evaluations and contributing to future evaluations.
Reluctance to share knowledge
- Competition, a lack of clarity in communication, and poor knowledge management/mobilisation led to a reluctance to share information, pool data, or drew on the evaluations of potential competitors.
- Evaluations often failed to use routine data and ignored the costs imposed on service providers and users.
- For commissioners of evaluations: Commissioners from multiple organisations should collaborate routinely to identify key cross-cutting challenges, jointly fund evaluations addressing common concerns, engage the evaluation community earlier when developing an evaluation specification and ensure budgets and timelines match the scale of the question(s) asked.
- For evaluators: Evaluators should engage earlier with commissioners and better balance competitive with collaborative behaviours to inform commissioners, develop standardised data sets, and make better use of routine data.
- For service providers: Service providers can be a great source of tacit knowledge, but they often experience evaluations as something to be endured rather than as something to shape. Providers should work with evaluators to develop better routine monitoring and data collection.
- For national policymakers in NGOs and government: Support the conditions under which the 'tragedy of the commons' can be avoided by: identifying and communicating priorities for evaluation research locally and nationally; monitoring the evaluation landscape; encourage and reward organisations for engaging with national priorities; and help nourish the 'commons' through sharing and collaborating while also ensuring adequate competition and variety.
Table of Contents
What is the problem?
What should be done differently?
Conclusions and recommendations
Responses from the warm-up question at each workshop
The research described in this report was funded by Age UK and conducted by RAND Europe.
This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
Permission is given to duplicate this electronic document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. For information on reprint and linking permissions, please visit the RAND Permissions page.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.