Getting to Outcomes Step 09. Continuous Quality Improvement (CQI)
This step provides a framework for using process and outcome evaluation data to make program improvements.
What Is This Step?
Step 9 will help you both review your past decisions and use your process and outcome evaluation data to determine what worked well, where there is room for improvement, and what changes may be needed the next time you run the program.
Why Is This Step Important?
CQI takes advantage of what you have learned over time from your evaluation to improve the program for the future without starting over from the beginning. It puts the investment made in evaluation to work by using the results to make changes and understand their effects for the next time you run the program. It helps all staff involved to keep your program fresh and a good fit for your participants, your organization, and your community. Although it began in manufacturing, CQI is becoming a part of routine operations in many health and social services organizations and can be applied across all programs, representing an emphasis by the organization on the quality of its services.
How Do I Carry Out This Step?
The CQI Review Tool will prompt you to summarize your evaluation data and work back through GTO Steps 1 to 8 as you assess what went well and what should be improved. You will evaluate whether you met the goal and desired outcomes you created in GTO Step 2 using the results from your process and outcome evaluations. This will prepare you to decide whether and how to revise your goals and desired outcomes, reassess fit and capacity, and revise your work plan for the next program implementation. The next step is to assess whether the changes you make will prove to be effective.
Tools Used in This Step
CQI Review Tool
The CQI Review Tool will help you create a snapshot of your program’s successes and shortcomings and prompt you to identify necessary improvements.
Instructions
- Make as many copies of the tool as necessary for you and your co-workers to complete this task. Try to include as many stakeholders as possible in this review.
- Assign a person responsible for collecting the completed GTO tools, including the notes containing all of the process and outcome evaluation data gathered in the course of the program. You also will need your program materials (e.g., manual) to help you complete this tool.
- Complete the first section: Priorities for Action.
- Use materials gathered and generated in GTO Step 1 (Needs and Resources Assessment) to enter your program’s information in the Targeted Need column. This could come from section 7 in the GTO Step 1 Triaging Among Needs Tool.
- Using the SMART Desired Outcomes Tool from GTO Step 2 (Goals and Desired Outcomes), enter the desired outcome statement associated with each need.
- Use outcome evaluation data.
- From your interpretation of the results in your Outcome Evaluation Summary Tool (GTO Step 8), check the impact on each of the SMART desired outcomes (reached, missed, or exceeded). Finally, determine whether any further action is needed (yes or no). Further action may be needed if you did not reach your desired outcome or if you believe there is room for improvement.
- Complete the second section: Process Evaluation.
- Using your GTO Step 7 Process Evaluation Summary Tool, complete the fields describing program dates and target population. The lettered fields are asking for (A) total target population, (B) total number of participants who attended at least one session, (C) total number of participants who attended every session, and (D) total number of participants included in the process evaluation. You can then calculate the percentage of your target population that actually attended (D divided by A) and then the percentage of actual participants included in the evaluation (D divided by B).
- Complete the third section: Planning Program Improvements.
- Once you know more about what has worked and not worked, you can make decisions about changes to make before the program is implemented again. The questions in this section prompt you to review past GTO steps. If your answers suggest that changes are needed, you might need to rework tools from previous steps. For example, if you need to change your goal or desired outcomes, you may need to make changes to the scales in your Outcome Evaluation Survey Tool. If you decide you need to make changes in any GTO step, go back and update the relevant GTO tool. Answer each of the questions honestly, and, where needed, create strategies for improvement for your next implementation.
- For more information on making a small change and understanding whether it makes a difference, review Promoting Success: A Getting To Outcomes Guide to Implementing Continuous Quality Improvement for Community Service Organizations, by Sarah B. Hunter, Pat Ebener, Matthew Chinman, Allison J. Ober, and Christina Y. Huang, Santa Monica, Calif.: RAND Corporation, TL-179-NIDA, 2015 . It provides more-specific guidance on conducting continuous improvement.
Example
- Completed by: Project team/manager
- Date: September
- Program: ROAD-MAP
1. Prioities for Action
| Targeted Need | Desired Outcome | Outcome Evaluation Result | Action Needed?
Yes/No |
|---|---|---|---|
|
More than 1/3 of older adults (age 50+) do not possess a 7-day supply emergency kit. |
To increase the number of program participants indicating that they possess a 7-day household emergency water supply by 20% from baseline to follow-up (3-month period). |
Progress on desired outcome.
|
No |
|
A high prevalence of older adults who take prescription medications do not possess emergency stockpiles. |
To increase the number of program participants indicating that they possess a 7-day supply of their medications by 15% from baseline to follow-up (3-month period). |
Progress on desired outcome.
|
Yes |
2. Process Evaluation
| Process Evaluation Types | Inputs | ||||||
|---|---|---|---|---|---|---|---|
|
Dates and Participation Targets |
Program dates: Eight ROAD-MAP workshops run between April 17 and May 15 |
||||||
|
A. Total target population: 20 per class, 8 classes = 160 participants |
|||||||
|
Target population characteristics:
|
|||||||
|
Program Adherence/Delivery |
B. Total program participants who attended at least one session: 115 C. Total who attended every session: N/A (program delivers entire curriculum in 1 session) |
||||||
|
What level of program adherence did you achieve (offer activities according to program requirements), and what evidence do you have to document this level of adherence? A program evaluator sat in on workshops and filled out a fidelity assessment tool based on the instructor’s delivery of the program content, methods, and implementation. The adherence was generally good, with trainers receiving score of “achieved” or “excelled” for the majority of the program content, methods, and implementation. Only 2 of the workshops had missing content. |
|||||||
|
Process Evaluation Results |
|
||||||
|
Evaluation participants (check all that apply):
|
|||||||
|
How well does the evaluation represent the population of participants in the program? (check one):
|
3. Planning Program Improvements
| Step-by-Step Review | Response | Changes for the Next Time? |
|---|---|---|
|
Did doing the program demonstrate the need for it? (GTO Step 1) How relevant and current to our participants are the needs data? Does an evaluation show participation, satisfaction, and results? |
The data are relevant to senior needs in the United States. Participant pre-program data also show need. |
Continue to monitor pre-program data for needs of the target population. |
|
Do we need to change goals and desired outcomes or potential participants? (GTO Step 2) Target different conditions or behaviors? Reset benchmarks up or down? |
We don’t need to change the current goals and desired outcomes because they are important targets for getting individuals prepared for disasters and they address needs that are specific to older adults. |
We will consider adding a third goal related to emergency communication plans. Primarily, we need to devote more time to the prescription medications supply goal. |
|
Should we consider another program? (GTO Step 3) Or are there other improvements we need to make? |
ROAD-MAP remains a relevant program to adapt for CEP aimed at older adults. |
N/A |
|
Does the program still philosophically and logistically fit our organization, community, and participants? (GTO Step 4) If not, why not? What adaptations could be made? Were any adaptions made? How did that go? |
Yes |
N/A |
|
Do we have the resources and capacities to do the program well? (GTO Step 5) Has there been a shift in resources? Are new staff capacities needed? |
Changes in residents will require that we involve/train new peer volunteers to help execute the program. |
Introduce staff to program by going over the program materials and GTO tool worksheets we completed to originally plan the program. |
Step Checklist
When you finish working on this step, you should have:
- Completed the Step 9 tool
- Documented successful program activities
- Assessed which program activities did not work well overall or for specific groups
- Identified areas for improvement
- Created strategies for improvement
- Increased buy-in within your organization by soliciting and acting on the suggestions of program staff