Evaluating the Q Initiative for UK Healthcare Quality Improvement

The aim of the evaluation was to inform the direction and strategy (including informing a future monitoring and evaluation framework and the data required to support future learning) of the Q initiative through providing real-time feedback throughout the first year of Q. The evaluation focused on identifying problems and practical solutions.

Objective 1: Design process

• Tell the story of the design process during the first year of Q, to provide an independent account of how the initiative evolved.
• Provide continuous and final recommendations for how the Q design process could be improved, how design processes could operate in subsequent years, and if the Health Foundation were to take a similar role in designing future initiatives.

Objective 2: Potential for Q

• Provide an independent account of the first year of Q, including identifying who was selected and how diverse they were, what they learned and how the network of participants functioned to support learning and change.
• Assess how effective the recruitment process for the first cohort was, including the internal management of recruitment and the devolved nominations process, look at the strengths and weaknesses of the selection processes used by each nominating organisation, and the diversity and suitability of participants produced by those processes.
• Work flexibly with the Q project team, sharing learning and engaging regularly in identifying emergent roles and barriers to progress.
• Carry out an end-of-year assessment of which parts of the theory of change are more and less valid, suggest amendments, and analyse whether Q is and/or is likely to be effective at executing it.
• Articulate specific options for the further design and implementation of Q during the first year and identify proposals for the initiative following the first year.
• Outline a monitoring, evaluation and quantitative and qualitative data collection strategy for the initiative to take forward following the first year.

The evaluation drew on both quantitative and qualitative data collection, including methods such as:

• Semi-structured ‘before’ and ‘after’ interviews with participants
• ‘Before’ and ‘after’ surveys of participants, including social network analysis
• Three rounds of focus groups with participants at the design events
• Document review of paperwork submitted to the Health Foundation by organisations who nominated participants
• After-action reviews with the Q project team
• Survey of organisations that employ or host participants

Context: Q members reported a range of views on what quality improvement is, what the barriers to its effectiveness are, and what they expect from Q

Members conceptualised quality improvement as a spectrum from a mind set to a discipline in its own right. Members reported that they face significant barriers in implementing quality improvement, principally lack of time and financial constraints. Interviewees were optimistic that there was benefit to be gained from the creation of a national initiative.

Implementation of a co-design process: The size of the founding cohort made co-design challenging but catalysed member buy-in

The large size of the cohort limited the extent to which all members could engage in the co-design process. Members felt that the events were well organised and that the products and brand were of high quality. The levels of commitment and loyalty expressed by most Q members interviewed at the end of the process were high.

Mechanisms of change: Q has connected founding cohort members

Members provided a multitude of positive examples of the beneficial impact that participating in the Q founding cohort had had on the ‘connecting’ and ‘developing’ strands of the central part of the theory of change.

Outcomes: Q is a promising but unproven initiative

Q founding cohort members and project team members agreed that Q did not make as much progress during the first phase as might have been hoped. The vast majority of interviewees stated that they would like to remain involved with Q in some capacity.

The next phase of Q should:

• Focus early on bringing substance to Q’s features
• Make clear the relationship between leadership and membership
• Avoid confusion over the recruitment strategy for members
• Stick with the current theory of change for now
• Harness diversity through a shared core of values supporting a wide variety of activities
• Transition from a formative to a summative evaluation

Future initiatives similar to Q should:

• Be clear about the founding cohort’s role and recruit accordingly
• Ensure that events and activities fit within an overall vision
• Make sure that accountability and processes for decision making are clear
• Maintain the strengths of Q