Toolkit: Building Resilience in Older Adults: Step 04. Evaluating and Improving Efforts to Promote Older Adults' Resilience

This step provides guidance on how to conduct an evaluation of your efforts, measure outcomes, and use these results to improve ongoing work.

What Is This Step?

This Step provides some guidance on how to conduct an evaluation of your efforts to promote older adults' resilience (Worksheet 4.1 and Worksheet 4.2). The Step then walks through ways to measure potential outcomes from your group's independent and collaborative work promoting older adults' resilience (Table 4.1 and Table 4.2). The Step concludes with guidance on how to use evaluation findings to inform and improve your ongoing work (Checklist 4.1 and Worksheet 4.3). This is a brief evaluation primer. More information with detailed evaluation and continuous quality improvement guidance is available online in RAND's Getting To Outcomes manuals:

Conducting an Evaluation

A design is the outcome evaluation term for the type of evaluation you will conduct. The type of design guides when you collect data and from which groups. For example, a simple and inexpensive design uses a questionnaire to collect data from older adults participating in a resilience activity just before you begin and after you complete the activity (often called a pre/post). Another type of design called the pre/post with comparison group compares participating older adults with a similar group of older adults not participating in the activity during the same time period. This way, you can be sure that any changes observed in the older adults participating in the activity from pre to post were real and were not happening to all older adults (i.e., if both groups improve the same amount, then the activity did not have an effect). This evaluation design is a more rigorous way to evaluate whether the activity achieved the desired effects. However, this design is more complicated, so you may want to consult a program evaluator.

Worksheet 4.1 provides some reflection questions to consider when designing an evaluation. Answering questions about the intended timing and audience of the evaluation, and your group's evaluation expertise and available evaluation resources, is important to informing your evaluation plan. Timing of the evaluation is important—it is best to begin an evaluation prior to conducting any activity so that you can collect some data from participants prior to their involvement with the activity. Requirements of grant funding, such as the Public Health Emergency Preparedness Cooperative Agreement, or deadlines for reporting to funders, boards, or other oversight bodies may also drive evaluation timing. Answering the questions in the "Timing of Evaluation and Intended Audience" section will help inform your decisions about evaluation timing. Finally, all evaluations require resources (e.g., staff expertise to conduct the evaluation, money for a survey). Answering the questions in the "Evaluation Expertise and Available Evaluation Resources" section will help you understand the resources your group has and might need in order to conduct an evaluation. If your group does not have evaluation expertise or established relationships with organizations that can provide evaluation expertise, you might consider engaging an external evaluator to help support your evaluation efforts. The American Evaluation Association provides a searchable database of members available for evaluation consulting. It is important to seek evaluation expertise while planning an evaluation.

Worksheet 4.2 provides an updated planning template (updated from Worksheet 3.4) in which you can include a brief summary of your evaluation plan for each activity.

Outcome and Process Measures

There are a variety of measures that can be used to capture older adults' resilience. We highlight a select set of measures that we used in a recent study of villages’ impact on older adults' resilience (see journal article manuscript, available from the authors). These measures capture factors related to older adults’ resilience in four areas: disaster preparedness, physical health, emotional health, and social health (Table 4.1). When selecting specific measures, you must consider how these measures align with activities (e.g., do you expect a change in these measures based on the specific activities your group is performing?). For example, if your group is providing targeted educational materials to improve older adults' disaster preparedness, then you could use the disaster preparedness measures. See Table 4.1 for examples of the types of activities that you could use each measure to evaluate.

In addition to aligning measures with the specific activities, you will also need to consider how the measures will be collected (timing, frequency, and person responsible), how the data from these measures will be analyzed, and what resources and expertise will be needed to understand and apply any findings. Collecting these outcome measures will allow you to answer important questions about whether your work is having the desired effect.

Table 4.1. Survey Questions to Measure Older Adults' Resilience

Disaster Preparedness

Sample Activity Measurement Domain Questions Response Options References
Disseminate targeted preparedness educational materials to older adults Preparedness knowledge
  • I am knowledgeable about local emergency plans for my community
  • I know the evacuation route to take in the event of an emergency
  • I know how to get information in an emergency
  • I know what supplies I need to securely seek shelter for up to 72 hours
  • I could help my neighbor, if he or she needed it, during a disaster
Strongly agree to strongly disagree Chandra, Williams, et al., 2013
Train older adults in emergency preparedness planning or processes Preparedness behaviors

In the past 12 months I have…

  • Participated in a neighborhood or community meeting about emergency preparedness
  • Been trained in how to help my neighborhood or my neighbor in responding to an emergency
  • Put together a household preparedness kit
  • Worked with people in my neighborhood to develop a community emergency plan (e.g., call-down lists, storing resources)
  • Attended training in psychological first aid or other type of training related to dealing with emotional stress of disasters
  • Identified where individuals who need extra help in a disaster may live
  • Put together a 3-day supply of prescription medications to use during an emergency
  • Signed up to be part of a Smart911 program
  • Signed up to receive government alerts during an emergency
Yes, No, Don't know Chandra, Williams, et al., 2013
Host an event where older adults can meet supporting organizations' representatives in their community Local supports
  • Do you belong to a community organization (e.g., school, church or other faith community, or volunteer organization) that you can depend on in a disaster?
  • Could you call upon one of your neighbors to assist you in an emergency, such as providing food, transportation, or help with your children?
Yes, No, Don't know Chandra, Williams, et al., 2013

Physical health

Sample Activity Measurement Domain Questions Response Options References
Conduct a training on the importance of and process for accessing primary care providers to maintain health Access to care Is there a place you USUALLY go when you are sick or need advice about your health? Yes, There is NO place, There is MORE THAN ONE place, Refused, Don't know Blewett et al., 2008
Access to care What kind of place do you USUALLY go when you are sick or need advice about your health? Don't get care anywhere, Clinic or health center, Doctor's office or HMO, Hospital emergency department, Some other place, Don't go to one place most often, Refused Blewett et al., 2008
Access to care Is that [fill in response from question above] the same place you USUALLY go when you need routine or preventive care, such as a physical examination or check up? Yes, No, Refused, Don't know Blewett et al., 2008
Recent care visit About how long has it been since you last saw or talked to a doctor or other health care professional about your own health? Include doctors seen while a patient in a hospital Never, 6 months or less, More than 6 months but not more than 1 year ago, More than 1 year but not more than 2 years ago, More than 2 years but not more than 5 years ago, More than 5 years ago, Refused, Don't know Blewett et al., 2008

Emotional health

Sample Activity Measurement Domain Questions Response Options References
Deliver an educational program on emotion and coping Active coping

These questions ask about what YOU usually do when YOU experience a stressful event.

  • I concentrate my efforts on doing something about it
  • I take additional action to try to get rid of the problem
  • I take direct action to get around the problem
  • I do what has to be done, one step at a time
Strongly agree, Moderately agree, Neither agree nor disagree, Moderately disagree, Strongly disagree Diener et al., 2010
Emotional well-being
  • I lead a purposeful and meaningful life
  • My social relationships are supportive and rewarding
  • I am engaged and interested in my daily activities
  • I actively contribute to the happiness and well-being of others
  • I am competent and capable in the activities that are important to me
  • I am a good person and live a good life
  • My material life (income, housing, etc.) is sufficient for my needs
  • I generally trust others and feel part of my community
  • I am satisfied with my religious or spiritual life
  • I am optimistic about the future
  • I have no addictions, such as to alcohol, illicit drugs, or gambling
  • People respect me
Strongly agree, Moderately agree, Neither agree nor disagree, Moderately disagree, Strongly disagree Diener et al., 2010

Social health

Sample Activity Measurement Domain Questions Response Options References
Conduct social engagement efforts where older adults get to know their neighbors and each other Social disconnectedness Approximately how many people do you know with whom you can discuss important matters? None, One or two, Three to five, Six to ten, More than ten, Don't know Suzman, 2009
Social disconnectedness

In the past two months…

  • about how often did you talk with one or more of these individuals (by phone, email, or in person)?
Several times a week, About once a week, About once a month, Less than once a month, Never Suzman, 2009
Social disconnectedness Approximately how many friends would you say you have? None, One or two, Three to five, Six to ten, More than ten, Don't know Suzman, 2009
Social disconnectedness

In the past two months…

  • about how often did you get together socially with friends or neighbors?
  • how often did you attend meetings of any organized group? (such as a choir, a committee or board, a support group, a sports or exercise group, a hobby group, or a professional society)
  • how often did you do volunteer work for religious, charitable, political, health-related, or other organizations?
Several times a week, About once a week, About once a month, Less than once a month, Never Suzman, 2009
Social isolation

In the past two months…

  • how often did you feel that you lacked companionship?
  • how often did you feel left out?
  • how often did you feel isolated from others?
  • How often do you feel that you can open up to other people about personal concerns?
  • How often do you feel that you can rely on other people to provide help when you need it?
Hardly ever (or never), Some of the time, Often Suzman, 2009

In addition to these outcomes that tell you something about older adults' resilience, your group might also want to collect process evaluation data. Process evaluations are designed to document and analyze the development and actual implementation of programs and other activities assessing whether and how well services are delivered as intended or planned. Process data can include tracking attendance or participation, participant demographics, participant satisfaction, and measures of implementation activities (for example, program fidelity measures, such as adherence to the program curriculum). Collecting process data can help your group answer important questions about the implementation of your activity:

  • How much of the activity did participants take part in?
  • What are the characteristics of participants?
  • How satisfied are participants? How satisfied are the staff who implemented the activity?
  • Was the activity implemented as planned?

These questions may provide explanatory support indicating why your activities may or may not have achieved their desired effects. If you need more detail, Getting To Outcomes provides a primer on process evaluation.

Partnership Measures

Given the importance of collaboration between villages, AFCs, and public health departments to achieve older adults' resilience, we also include several measures of partnership that may help provide insight about the presence and quality of these partnerships. These measures reflect a brief version of a longer tool called PARTNER that has been used to capture the work of public health collaboratives across the United States.

To understand how well a partnership is working, data are needed to describe:

  • Who is part of the partnership and how frequently they work together
  • Strength and quality of interactions
  • Level of trust and value within the partnership
  • Changes in collaborative activity over time (captured through repeating measures over time)
  • Organizational and community benefits of collaborative activities

Table 4.2 provides sample questions that can be used to evaluate the different aspects of partnership. Special analysis skills are required to understand how partnerships result in organizational networks and to calculate network level measures. Technical assistance using PARTNER measures is available through the Center on Network Science at the University of Colorado Denver. The CDC also has a guide that describes the fundamentals of evaluating partnerships. It includes tools to help plan a partnership evaluation and contains a simple inventory that can be used to self-assess a partnership.

In addition to capturing partnership dynamics, you may want to capture the impact of collaborative activities. Table 4.3 contains sample measures that could be used to describe the success of the collaborative activities outlined in Step Three.

Table 4.2. Survey Questions to Measure Partnership

Measurement Domain Questions
Presence of partnership Over the past month, what organizations or agencies have you worked most closely with to promote older adults' resilience?
Questions asked for each organization listed in the response to the presence of partnership question
Frequency of interaction About how frequently have you communicated with [insert organization name] in the past month?
Level of influence How much power or influence (e.g., decisionmaking authority, leadership responsibility) do you think [insert organization name] has over activities to promote older adults' resilience? (Select one and please take your best guess)
Level of involvement How involved is [insert organization name] in promoting older adults' resilience?
Resource contributions To what degree has [insert organization name] contributed resources to promoting older adults' resilience?
Reliability To what degree has [insert organization name] been reliable in promoting older adults' resilience?
Open Communication To what degree is [insert organization name]'s communication open and transparent (for example, their purpose and what they intend to do are clear) about promoting older adults' resilience?
Summary of questions asked of a single organization
Organizational benefits What benefits has your organization received as a result of working with each organization?
Community benefits In your opinion, what has been the impact of your and your partners work promoting older adults' resilience?

Table 4.3. Sample Measures for Collaborative Activities

Collaborative Activity Sample Measure
Encourage older adults to join emergency information systems Proportion of older adults in the community who are signed up for automated emergency alert and information systems
Participate in preparedness planning to ensure that the needs of older adults are represented Presence of specific guidance in community AFC and preparedness plans that account for the needs and strengths of older adults
Develop/provide feedback on concise targeted educational materials for older adults Penetration of dissemination efforts and uptake of information
Train or educate each other on specific areas of expertise (e.g., older adults, emergency preparedness) Proportion of supportive service agencies that have response plans, communication plans, and continuity-of-operations plans in place to assist older adults during an emergency
Encourage older adults to join community emergency response teams or Medical Reserve Corps Presence of older adults in community emergency response teams or Medical Reserve Corps

Using Evaluation Data to Inform Your Work

Once your evaluation data are collected and analyzed, your group will be able to determine whether its work is having the desired effects. It is important to acknowledge that the more rigorous your evaluation design is, the more confidence you can have that your activity produced or did not produce the intended effects. Worksheet 4.3 provides a template to document which outcomes your activity was successful in achieving and which may require further action. In the first column, list each outcome your group was tracking (for example, disaster preparedness knowledge or strength of partnership interaction). Then describe in a single sentence any difference or change in the outcome (second column), and use the third column to indicate how this changed from before the activity was conducted (whether it got better, got worse, or stayed the same). In the fourth column, specify whether this met your expectations for the activity. For example, if the activity was supposed to improve disaster preparedness knowledge and your evaluation showed that knowledge did not improve after the activity, you would mark "Same" in the third column and "No" in the fourth column. In the fifth column, specify any action needed. For example, if the activity was intended to strengthen partnership interactions but you missed your expectations, you would mark "Yes" to indicate that action is needed. Finally, the last column asks you to reflect on any potential barriers that might have influenced whether the activity had the desired effect (for example, participation in the activity was low).

Checklist 4.1 walks you through a brief continuous quality improvement exercise to help identify potential challenges that might have affected your activity and specific actions that you could take to address these challenges moving forward. Then record who will participate in the action, the resources needed, location details, and the target date for improvement as new activities in your existing plan (Worksheet 4.2). Enhancing your plan with improvement activities using Worksheet 4.2 will help you identify what is necessary to achieve your goals and help you specify a target date for improvements to be made. If possible, complete the improvements prior to doing the activity again.

Tools Used in This Step

Worksheet 4.1. Issues to Consider for My Evaluation

Worksheet 4.1. Issues to Consider for My Evaluation

Jump to

Instructions

Worksheet 4.2. Updated Plan for Short-Term and Long-Term Resilience Activities

Worksheet 4.2. Updated Plan for Short-Term and Long-Term Resilience Activities

Jump to

Instructions

  1. Next to "Sample," specify the target population and estimated number of program participants for the evaluation.
  2. Next to "Data Collection," specify the timing of your evaluation (the dates the evaluation will occur, which should be tied to the beginning and end of the activity).
  3. Next to "Measures," specify any measures you plan to use. These could include the process or outcome measures described in Table 4.1 and Table 4.2.
  4. Be sure to update the "Where Will We Get Any Resources We Need?" column to include any expertise, supplies, or equipment (for example, access to computers); staff time; financial resources (for example, money to support an online survey subscription); and organization resources (for example, buy-in from leadership) that might be needed for the evaluation.
Download

Worksheet 4.3. Review Program Outcomes

Worksheet 4.3. Review Program Outcomes

Jump to

Instructions

Checklist 4.1. What Continuous Quality Improvement Actions Are Needed to Improve the Activity?

When you finish working on this step, you should have:

  • Completed the Step 4 tools for all programs under consideration
  • Developed an understanding of what fit means
  • Considered the most important aspects of your program to make sure there is a good fit with your target population, your organization, and your community
  • Determined the right adaptation needed, if any, to improve the fit of your program(s)
  • Further narrowed your choice of programs to implement

Summary

This Step provided information on process, outcome, and partnership measures (Table 4.1 and Table 4.2) and guidance on how to use those measures as part of an evaluation of your activities to promote older adults resilience (Worksheet 4.1 and Worksheet 4.2). This Step also provided guidance on how to use the findings from your evaluation to improve and inform your work going forward (Checklist 4.1 and Worksheet 4.3).

After using this Step, you should have selected your evaluation strategy, planned how to use your evaluation data, and reflected on what your evaluation findings mean for your current and future work promoting older adults' resilience.

The content and worksheets of this toolkit are intended to be reused, even after you have completed your evaluation. Consider reviewing the toolkit annually to continue improving your work and refining your evaluation.

Nice job completing the toolkit!