The Mental Health Association of San Francisco founded the Center for Dignity, Recovery, and Empowerment (the Center) with funding provided through the Stigma and Discrimination Reduction (SDR) initiative of the California Mental Health Services Authority (CalMHSA) Prevention and Early Intervention (PEI) project. The Center was funded jointly under two SDR programs: the Values, Practices, and Policy Resource Development program and the Promising Practices program. As part of the work conducted under the Resource Development program, the Center aimed to develop and make available resources to promote mental illness–stigma reduction programs. As part of this effort, staff from the Center provided technical assistance (TA) for stigma-reduction programs in the form of individualized support, formal training, and/or program-evaluation resources.
The Center provided TA to community-partner organizations—that is, community-based organizations, mental health–services agencies, and county mental health departments that each housed a mental illness–stigma reduction program. In particular, the TA effort was targeted at community-partner organizations whose program staff were interested in or were currently using contact-based strategies (i.e., strategies that involve having people who have experienced mental health challenges tell their stories of illness and recovery) for reducing mental-illness stigma.
The Center reported that its staff recruited most community partners at advocacy meetings and through their existing connections at community-based organizations, although some community partners got involved by approaching the Center staff after Center-hosted events (e.g., its Tools for Change conference, community outreach events) (Jackson, Martinez, and Pham, 2014). Most of these organizations were trying to initiate or improve a speakers bureau program (i.e., a collection of people who have experienced mental health challenges and who are willing to speak to others about their experience). Contact-based strategies have proven to be effective in reducing stigmatizing attitudes and beliefs toward people with mental health challenges (Corrigan et al., 2012; Griffiths et al., 2014).
All participating community-partner organizations received TA (i.e., support to help community-partner organizations exe-cute their efforts; Mitchell, Florin, and Stevenson, 2002). Though research on TA processes and outcomes is limited (Chinman, Hannah, et al., 2005), some research supports the premise that the provision of TA to community organizations can successfully build their capacity to deliver program activities (e.g., Chinman, Early, et al., 2004; Chinman, Ebener, et al., 2014). At a minimum, all community partners involved in the TA process received a set of documents that focused on stigma-reduction program messaging (e.g., guidelines on developing and delivering effective personal-recovery narratives), program design (e.g., essential features of stigma-reduction programs), program evaluation (e.g., fidelity and implementation assessments), staffing (e.g., interviewing potential speakers), and targeting stigma messages (e.g., how to tailor messages to specific audiences).
In addition to receiving the set of documents, community partners received varying levels of TA to address their organization's needs related to their contact-based stigma-reduction programs. Thus, community partners were not randomly assigned to receive a specific type of TA; they partnered with the Center to determine the type of TA that both parties agreed would be helpful. To facilitate sampling for our evaluation, community partners were sorted into two groups based on the type of TA that the Center reported providing to the partners: one receiving individualized support and training and the other receiving program evaluation.
Community partners in the individualized support/training group received one-on-one TA from the Center staff and/or formal training on how speakers can best tell their stories of mental illness and recovery. Community partners in the program-evaluation group may also have received individualized support and training, but the distinguishing feature is that for these partners, the Center staff formally evaluated one of the community partners' contact-based stigma-reduction programs using the California Quality Improvement Fidelity Assessment Implementation Ratings (CQI-FAIR) tool developed by the Center. The CQI-FAIR allows for evaluation on five main domains: program design, appropriate targeting for the audience, staffing, messaging, and follow up and evaluation. The Center subsequently provided programs undergoing evaluation with a report summarizing the results.
This article describes the results of RAND's evaluation of the Center's TA process. The evaluation was conducted by RAND and funded by CalMHSA and involved semistructured interviews with community partners who participated in the Mental Health Association of San Francisco TA process. Because the organizations varied on multiple dimensions (e.g., target audiences, geographic location), and the support they received was tailored to their specific needs, we focused on determining community partners' organizational goals and the extent to which the Center helped them meet these goals. For example, a community partner initiating a new SDR program would likely have different needs than a partner with an existing SDR program who is attempting to build capacity. A community partner initiating a new program may require information about how to recruit and train new speakers, whereas community partners with existing programs may be more interested in evaluating their program's outreach processes and outcomes and in receiving individualized support to improve current processes.
As a result, we sampled a diverse set of community partners and aimed to represent the experiences of two groups of community partners: those receiving individualized support or training (and no program evaluation) as well as those partners undergoing program evaluation. The interview assessed satisfaction with, and the impact of, the Center's efforts from the perspective of community partners and attempted to identify particular strengths and weaknesses to inform future efforts. Because the focus of the effort was on contact-based strategies for stigma reduction, we assessed the extent to which use of contact was influenced by the Center efforts.
The evaluation relied on qualitative methods, specifically semi-structured interviews with representatives from community-partner organizations who engaged with the Center staff as part of the resource-development TA process. We conducted semi-structured interviews with 15 (about one-half) of the Center's resource-development community partners between August and October 2014.
Community Partner Characteristics
In June 2014, the Mental Health Association of San Francisco provided us with a list of 27 community partners who participated in the resource-development TA process. The Mental Health Association of San Francisco reported that it recruited community partners through both direct outreach and networking at community-advocacy events, though some community partners approached the Mental Health Association of San Francisco to participate. This list of community partners also included information about what form of TA the Center provided to each community partner (i.e., individualized support or training, program evaluation), whether the partners were likely to have targeted racial/ethnic minorities or rural communities, whether the SDR program was new or previously existing, the California region in which the partner is primarily located, and information about the primary point of contact at the organization.
We used this information to divide the partners into two groups: those whose programs had been evaluated in collaboration with the Center (n = 13) and those who received individualized support and/or training resources (but not program evaluation) from the Center (n = 14). We drew a random sample of eight organizations from each of these two groups to ensure that we adequately represented organizations using each of the two main types of TA provided. During recruitment, representatives of some sampled organizations were unavailable. These organizations were replaced with others with similar characteristics (e.g., new or existing SDR program, served minority or rural populations), where possible.
The final sample (see Table 1) consisted of 15 community partners: seven received individualized support and/or training from the Center, and eight received program evaluation. Relative to the full set of community partners who received individualized support and/or training, the sampled group comprised fewer partners who served specific racial/ethnic communities, more partners who served rural communities, and fewer existing SDR programs (versus those starting a new program). Compared with the full set of community partners who received program evaluation, the sampled group was fairly similar in composition.
Table 1. Characteristics of Mental Health Association of San Francisco TA Community Partner Sample (n = 15) and Full Set of TA Community Partners (n = 27)
|Percentage of Community Partners Serving a Specific Racial/Ethnic Community||Percentage of Community Partners Serving Rural Communities||Percentage of Community Partners with an Existing SDR Program|
|Sampled participants who received individualized support/training only (n = 7)||29||57||14|
|All participants who had received individualized support/training only (n = 14)||43||43||29|
|Sampled participants who received program evaluation (n = 8)||25||25||88|
|All partners who had received program evaluation (n = 13)||23||23||77|
NOTE: Data in this table were provided by the Mental Health Association of San Francisco (Jackson, 2014).
Characteristics of Sampled Community Partners' Contact-Based Stigma-Reduction Programs
All of the community partners in the sample worked with the Center either to initiate or enhance a contact-based stigma-reduction program. Moreover, for all but one community partner, the TA process focused on speakers bureau programs, consisting of having individuals with their own experiences with mental illness and recovery talk about their story in person. Three of the programs also used additional means, such as showing videos, to foster contact with individuals with mental health conditions. Eight of the stigma-reduction programs existed prior to the TA, and seven were new. The frequency of contact-based presentations made by community partners with speakers bureaus varied. Some of the partners who had recently developed a speakers bureau did not exist long enough at the time of interview to make an estimate of presentation frequency. Other estimates ranged from a low of once or twice a month to a high of 12 to 13 presentations per month.
The interview protocol was developed by the RAND SDR evaluation team and covered a variety of topic areas (see Table 2), including community-partner goals for engaging in the TA process and whether these were met, perceived benefits of TA participation, and satisfaction with the TA process. To help us understand and interpret those responses, we also assessed SDR program characteristics (e.g., new or existing program). The interviews also touched on the perceived cultural competence of the TA process.
Table 2. Interview Topics and Sample Questions
|Topic||Sample Interview Questions|
|Partner goals for TA process||What were your goals in working with the Center? To what extent have you achieved these goals while working with the Center? Would you say your goals were met wholly, partially, or not at all?|
|Perceived benefits of TA||What was the most important information that you received from the Center regarding delivering a stigma-reduction program? What was the most important thing you learned about your stigma-reduction program as part of the evaluation? What was the most important evaluation-related tool that you received from the Center?|
|Satisfaction with TA process||Do you feel the Center staff understand your organization and its goals for reducing stigma? Would you say that yes, they understand; they sort of understand; or no, they don't understand? Was there anything that made it difficult to work with the Center? Were there things you really liked about working with the Center?|
|SDR program characteristics||Are your stigma-reduction efforts contact based? Are you still delivering a stigma-reduction program? How often?|
|Cultural competence||Do you feel that the information and tools provided by the Center were appropriate for your target audience? Would you say that yes, they were appropriate; they were sort of appropriate; or no, they were not appropriate?|
The interview began by asking community partners about their goals for working with the Center and the extent to which their goals were achieved. Participants who received individualized support and/or training from the Center were asked to describe how much they already knew about stigma-reduction programs prior to working with the Center and to comment on which resources provided by the Center were the most important or useful. They also indicated whether they had considered or were using other stigma-reduction approaches. Participants whose programs were evaluated by the Center were asked similar questions, including about baseline knowledge of program evaluation, resources provided, which resources proved most useful or important, and whether other evaluation tools or strategies were used or considered. Participants answered a series of questions about their interactions, both positive and negative, with Center staff. Participants indicated whether they targeted specific audiences for stigma reduction (e.g., racial/ethnic minority communities, youth) and whether they thought the resources provided by the Center were appropriate for their target audience. Finally, participants provided information about the characteristics of their stigma-reduction programs.
In this section, we discuss the results of the interviews in terms of community partners' goals in working with the Center and attainment of those goals, community-partner perceptions of resources and information provided by the Center, and community-partner perceptions of the Center staff.
Community Partner Goals
When asked to reflect on their goals for engaging in the TA process, community partners expressed a variety of goals focused around several themes:
- Improve stigma-reduction presentations.
- Strengthen stigma-reduction messages in existing speakers' presentations.
- Tailor stigma-reduction messages to different audiences based on culture, age, sexual orientation, etc.
- Enhance professionalism of speakers bureau presentations.
- Launch a new speakers bureau to promote stigma reduction.
- Increase capacity of an existing speakers bureau to reduce stigma.
- Train and add more speakers.
- Expand number of presentations made.
- Evaluate efforts and determine areas for improvement.
- Establish metrics and better understand effectiveness of efforts.
- Provide feedback on speakers' presentations.
Most Community Partners Felt That Their Goals Had Been Achieved
Overall, nine of the community partners felt that their goals had been wholly met, and four felt that their goal had been partially met. Only one partner felt that his or her goal was not met by the end of the TA process. One partner indicated that he or she had entered the process without an explicit goal and thus was unable to rate whether the organization's goals had been achieved.
Of the community partners who received program evaluation, six felt that their goals had been wholly met, and two felt they had been partially met. Of the community partners who partnered with the Mental Health Association of San Francisco for training or individualized support only, three felt that their goals had been wholly met, two felt that their goals had been partially met, and one felt that his or her goals had not been met. One partner did not respond because he or she did not have explicit goals upon entering the TA process.
Prior to the TA Process, Community Partners Differed in Their Self-Reported Knowledge of SDR Program Administration and Evaluation
All seven community partners who received individualized support or training only felt that, prior to entering the TA process, they had at least “some” knowledge of how to administer a stigma-reduction program (four partners) or “a lot” of knowledge (three partners). This is a contrast to community partners who underwent program evaluation. Of these community partners, four felt that, prior to the TA process, they had “some” knowledge of program evaluation, and only two felt that they had “a lot” of knowledge. Two partners felt that they had no knowledge of SDR program evaluation prior to working with the Center. These differences might explain or contribute to the differences in goal achievement just noted. It is possible that those with less prior knowledge and experience got more out of the partnership, but we are unable to explore this thoroughly given the small sample size.
Perceptions of Provided Information and Tools
Community Partners Showed Little Consensus on Which Pieces of Information and Tools Were Most Useful
There was no overall consensus among community partners on which tools or resources were most useful. Among community partners receiving individualized support and/or training, two community partners each identified a specific tool provided by the Center as being the most useful or important: the “What is stigma?” brief document and a copy of a speaker-training curriculum. Individual community partners also mentioned various other useful information, such as information about myths and facts related to mental illness and the information delivered as part of training (e.g., sources of stigma, strategies and interactive activities about how speakers can best tell their personal stories of illness and recovery).
Of the community partners undergoing program evaluation, all but one named specific resources provided by the Center as being among the most useful or important items received. These resources included the CQI-FAIR program evaluation tool (three partners), a pre-post survey for administration before and after speakers' sessions (two partners), the Platform Skills Fidelity Measure for evaluating speakers (one partner), a document providing tips for developing a strong message (one partner), and the individualized program-evaluation report developed by the Center for that specific program (two partners).
Individual community partners also indicated that beyond the resources provided, they also learned a variety of important lessons from the TA process. These include the need to allocate sufficient time to train speakers, the importance of providing feedback to speakers on their presentations, the importance of gaining better understanding of who is reached through speakers bureau presentations, the need to learn about the audience to whom speakers will be presenting prior to arrival, the value of implementing pre- and post-presentation surveys to understand changes in audience attitudes, and the importance of providing audience members with a “call to action” as part of the presentations.
Community Partners Felt That the Center's Efforts Were Appropriate for Diverse Audiences
Community partners were asked whether they felt that the materials and services provided by the Center were culturally appropriate for different possible target audiences (e.g., transition-age youth, different racial/ethnic communities). Ten of the partners felt that the information provided was appropriate for many audiences, and four reported that the information was at least “sort of” appropriate. One partner had not yet used the tools and thus did not provide a response. Five partners commented that although the resources were not targeted toward a specific audience, they felt that materials could be tailored to meet the needs of various target groups. One partner, however, did feel that the materials felt very technical and thus might not be appropriate for all audiences.
Perceptions of Interactions with the Center Staff
Community Partners Largely Felt That the Center Staff Understood Their Needs and Goals and Were Responsive to Them
Partners were asked how well they felt the Center staff understood their needs and goals and how responsive the Center staff were. Twelve partners felt that the Center staff understood their needs and goals, and three felt that the Center “sort of” understood. Twelve partners felt that the Center staff were responsive. Two partners felt that staff were “sort of” responsive to their requests, and another partner felt that the Center staff was not responsive.
Community Partners Felt That Some Staffing Issues Presented a Challenge to the TA Process
Eleven partners expressed a range of difficulties, while the other four partners reported no difficulties. Despite an overall sense that the Center staff was understanding and responsive, four partners expressed concerns about the Center's staffing. The following staffing concerns were expressed by community partners, but none of these were noted by more than one organization: feeling the Center was too far away to facilitate in-person interaction with the Center staff, limited availability of the Center staff for conducting training, frequent staff turnover, and difficulties scheduling training for convenient times. Other concerns not related to staffing (none raised by more than one organization) included: wanting more training, lacking shared understanding of the need for and goals of the training, capacity-building goals not being met, lack of follow through by the Center staff on providing requested information, too narrow of a focus on speakers bureaus (relative to other stigma-reduction approaches), and dislike of the evaluation approach taken by the Center. One partner mentioned that a training session held for his or her organization did not run smoothly.
Community Partners Viewed the Center Staff's Demeanor and Personal Experience with Mental Illness as Major Strengths
All community partners expressed positive aspects of working with the Center. Many partners felt that the Center staff had many positive characteristics, including being “nice,” “friendly,” “professional,” “supportive,” “responsive,” and accessible by phone and email. Others acknowledged the Center staff's commitment and expertise in stigma reduction. Several partners highlighted the Center staff's personal experience with mental illness and recovery as a strength, and one partner highlighted the cultural diversity of staff as another strength.
Most Community Partners Would Work with the Center Again
At the time of the interviews, 11 of the partners were still in contact with the Center. Twelve partners said that they would contact the Center in the future for assistance. One said he or she might but would possibly consider a local resource first, and another said he or she was unsure as his or her internal program focus had shifted. One partner indicated that he or she would not contact the Center in the future.
Conclusions and Recommendations
Overall, community partners reported positive experiences with the Center TA process. Most community partners felt that their goals of starting a new SDR speakers bureau program or enhancing an existing program were achieved. Community partners varied in their estimation of which tools and resources provided by the Center were most useful for either training/individualized support or program evaluation. Several community partners who underwent program evaluation found discussions of their evaluation results to be some of the most helpful information received. Most community partners felt that the information and resources provided by the Center were appropriate for diverse audiences (though not tailored for specific audiences).
Nearly all community partners reported that interactions with the Center staff were positive, though 11 community partners expressed some difficulties or challenges in partnering with the Center. Most community partners felt that the Center staff understood their needs and were responsive. They felt that staff members' demeanors were positive and viewed the fact that many of the Center staff have had their own experiences with mental illness as a strength. Though several partners felt that the Center staffing challenges made the TA process challenging at times, most partners said they would work with the Center again in the future if they needed assistance with their SDR program.
We provide the following recommendations for improving the TA process:
- Hold explicit, in-depth discussions with community partners about individualized support and training goals. Community partners who received individualized support or training (and no program evaluation) were slightly less likely to feel that their goals had been met by the TA process relative to those who underwent program evaluation. Having more structured discussions with community partners about the goals, processes, and expected outcomes of the individualized support or training process, in addition to adjusting the materials and information provided by the Center accordingly, may help ensure that all community partners' goals are met.
- Continue to provide a variety of resources to meet the varying needs of different community partners. The lack of consensus about which materials and information were most useful to community partners suggests that the Center's current strategy of providing a wide variety of information and resources for all phases of SDR program administration and evaluation is an appropriate one that should be continued.
- Use remote learning and webinar opportunities to overcome staffing challenges. Because several community partners expressed difficulties related to the Center staffing and availability, the Center should consider strategies that overcome these challenges. Examples may include conducting trainings remotely via videoconference or through webinar formats to overcome geographic distance and minimize the burden of travel time on staff. Webinars can be used again for other trainings in the future and may also have the positive side effect of contributing to the sustainability of the Center's SDR efforts.
We note several limitations of this evaluation. A single individual represented community-partner agencies and organizations, and his or her perspective does not necessarily reflect the myriad of those at his or her organization. Also, we conducted interviews with a small number of community partners, rendering it difficult to draw conclusions about how partners who vary across multiple dimensions (e.g., in target audiences of efforts, location within California) compare to one another. Community partners were not randomly assigned to different types of TA, and thus differences between the community partners receiving individualized support and training and those receiving program evaluation may be due in part to differences in the characteristics of organizations selecting these different types of TA. Should the efforts of the Center continue, a richer evaluation that includes all community partners and long-term follow up on the usefulness of TA provisions should be implemented.
The Center for Dignity, Recovery, and Empowerment, “Platform Skills Fidelity Measure,” no date. As of August 21, 2015:
The Center for Dignity, Recovery, and Empowerment, “Pre-Test: Complete Prior to Participating in the Program,” no date. As of August 31, 2015:
The Center for Dignity, Recovery, and Empowerment, “Toolkit for Community-Based Stigma-Reduction Programs,” no date. As of August 31, 2015:
Chinman, Matthew, Debee Early, Patricia Ebener, Sarah B. Hunter, Pamela Imm, Penny Jenkins, and Abraham Wandersman, “Getting to Outcomes: A Community-Based Participatory Approach to Preventive Interventions,” Journal of Interprofessional Care, Vol. 18, No. 4, January 2004, pp. 441–443.
Chinman, Matthew, Patricia Ebener, Q Burkhart, Karen Chan Osilla, Pamela Imm, Susan M. Paddock, and Patricia Ann Wright, “Evaluating the Impact of Getting To Outcomes—Underage Drinking on Prevention Capacity and Alcohol Merchant Attitudes and Selling Behaviors,” Prevention Science, Vol. 15, No. 4, August 2014, pp. 485–496.
Chinman, Matthew, Gordon Hannah, Abraham Wandersman, Patricia Ebener, Sarah B. Hunter, Pamela Imm, et al., “Developing a Community Science Research Agenda for Building Community Capacity for Effective Preventive Interventions,” American Journal of Community Psychology, Vol. 35, Nos. 3–4, June 2005, pp. 143–157.
Corrigan, Patrick W., Scott B. Morris, Patrick J. Michaels, Jennifer D. Rafacz, and Nicolas Rüsch, “Challenging the Public Stigma of Mental Illness: A Meta-Analysis of Outcome Studies,” Psychiatric Services, Vol. 63, No. 10, October 2012, pp. 963–973.
Corrigan, Patrick W., and John R. O'Shaughnessy, “What Is Stigma?” The Center for Dignity, Recovery, and Empowerment, 2007. As of August 31, 2015:
Corrigan, Patrick W., Eduardo Vega, Jon E. Larson, and the California CDSISE steering committee, “CQI-FAIR for SDR Programs: California Quality Improvement, Fidelity, Assessment, & Implementation Ratings for Stigma & Discrimination Reduction Programs,” The Center for Dignity, Recovery, and Empowerment, 2012. As of August 31, 2015:
Griffiths, Kathleen M., Bradley Carron-Arthur, Alison Parsons, and Russell Reid, “Effectiveness of Programs for Reducing the Stigma Associated with Mental Disorders: A Meta-Analysis of Randomized Controlled Trials,” World Psychiatry, Vol. 13, No. 2, June 2014, pp. 161–175.
Jackson, Valerie, email discussion with the first author on the Center's Resource Development activities, June 10, 2014.
Jackson, Valerie, Monica Martinez, and Khoi Pham, phone call with the first author on the Center's Resource Development activities, May 8, 2014.
Mitchell, Roger E., Paul Florin, and John F. Stevenson, “Supporting Community-Based Prevention and Health Promotion Initiatives: Developing Effective Technical Assistance Systems,” Health Education and Behavior, Vol. 29, No. 5, November 2002, pp. 620–639.
 The Center's efforts under the Promising Practices program are not addressed in this article.
 These tools are available online in the “Tools” section of http://www.dignityandrecoverycenter.org.
 The CQI-FAIR and other program evaluation tools are available at http://www.dignityandrecoverycenter.org/tool_type/program-evaluation.
 The original list contained 33 community partners. Six community partners, however, were deemed ineligible for participation in the evaluation because they did not receive TA from the Center (n = 3), no contact information was available (n = 2), or the program had been dissolved (n = 1).
 See Corrigan and O'Shaughnessy, 2007.
 See Corrigan, Vega, et al., 2012.
 See The Center for Dignity, Recovery, and Empowerment, “Pre-Test: Complete Prior to Participating in the Program,” no date.
 See The Center for Dignity, Recovery, and Empowerment, “Platform Skills Fidelity Measure,” no date.
 See The Center for Dignity, Recovery, and Empowerment, “Toolkit for Community-Based Stigma-Reduction Programs,” no date.
This research was conducted in RAND Health, a division of the RAND Corporation.