CalMHSA Stigma and Discrimination Reduction Online Resources

Reducing stigma and discrimination associated with mental illness is an important goal. According to Mental Health: A Report of the Surgeon General, stigma exerts negative effects by causing people to avoid “socializing or working with, renting to, or employing” individuals with mental health challenges, which results in decreased opportunities and resources and, ultimately, low self-esteem, isolation, and hopelessness (U.S. Department of Health and Human Services, 1999). Many individuals and organizations may lack the knowledge or resources on how best to address issues of stigma and discrimination. With funds from the Mental Health Services Act (Proposition 63), the California Mental Health Services Authority (CalMHSA) developed and implemented three statewide prevention and early intervention (PEI) initiatives that focus on mental illness stigma and discrimination reduction (SDR), suicide prevention, and student mental health.

Under the SDR initiative, PEI activities include a social marketing campaign, educational trainings, and online resources. Program partners developed a wide range of SDR online resources targeting individuals in positions to influence the lives of people with mental health challenges, known as “gatekeepers,” and the general public. Additionally, program partners developed resources to help individuals find information and support related to mental illness. As part of the CalMHSA statewide evaluation of the PEI initiatives, RAND evaluated the utilization of these online resources, which were distributed through websites created by program partners. This article provides an overview of the SDR-related online resources and data that describe the use of these websites.

Overview of Online Resource Development

As part of the CalMHSA SDR initiative, most program partners developed informational and online resources related to stigma and discrimination reduction for a variety of audiences. Resources are diverse and include fact sheets, toolkits, guidelines, reviews, discussion forums, a documentary, and short videos containing personal stories that aim to reduce stigma and discrimination (see Table 1 for a description of sites tracked). In this article, we focus on the websites or portions of websites that are created, hosted, and maintained with CalMHSA funds.

SDR Program Partner and URL	Website Description	Primary Target Audience	RAND Website Usage Tracking Period
Disability Rights California www.disabilityrightsca.org/CalMHSA	The Disability Rights California website contains a section featuring more than 30 antidiscrimination factsheets translated into many languages. Topics covered include definitions and effects of stigma and discrimination, mental health parity, and the prevention of discrimination as it relates to education, employment, housing, jails and prisons, and health and mental health care.	General public, gatekeepers	March 15, 2013, to June 30, 2014
Entertainment Industries Council, Inc. www.eiconline.org/teamup	TEAM Up (Tools for Entertainment and Media) is a set of online resources for journalists and entertainment media creators that was launched in June 2013. The resources cover mental health concerns and include the following: a style guide for reporting on mental health (“Style Guide: Reporting on Mental Health”); story ideas and interview tips; social media guidelines (“Social Media Guidelines for Mental Health Promotion and Suicide Prevention”); depiction suggestions; an email link for media creators to obtain First Draft technical assistance; videos of Entertainment Industries Council events; podcasts discussing mental illness stigma; links to mental health—related publications; English- and Spanish-language resources (e.g., “Muestra Esto”); and a calendar of CalMHSA events relevant to media professionals and journalists.	Journalists, entertainment media creators	June 8, 2013, to June 30, 2014
Integrated Behavioral Health Project ibhp.org	The “virtual library” on the Integrated Behavioral Health Project/Center for Care Innovations website contains resources to support integrated care among primary care, mental health, and substance abuse treatment sectors. The “Partners in Health: Mental Health, Primary Care and Substance Use Interagency Collaboration Tool Kit” (2nd Edition, 2013), is intended to support collaborations across primary care, behavioral health, and substance abuse treatment sectors. The 2009 edition of the toolkit incorporates content specific to the implementation of integrated care within the state of California and adds key resources related to the integration of care for substance use—related problems. The 353-page toolkit was posted for download on April 26, 2013. The toolkit contains many different types of information, including scientific research on integrated behavioral health, quotes from medical and behavioral health providers on the positive benefits of integrated care, illustrative examples from case studies, and practical information on the logistics of establishing interagency collaborations.	Professionals working in primary care, mental health services, and substance abuse treatment	June 8, 2013, to June 30, 2014
Mental Health Association of San Francisco dignityandrecoverycenter.org	The website for the Mental Health Association of San Francisco Center for Dignity, Recovery, and Empowerment hosts information about its Resource Development and Promising Practices programs. The site contains a directory of SDR programs, including stigma reduction, education and training, and culturally responsive programs. The organization has developed and made available many tools and resources under its Resource Development program around the development, maintenance, and evaluation of contact-based stigma reduction programs.	General public, gatekeepers, organizations seeking to implement speakers bureaus	June 8, 2013, to June 30, 2014
Runyon Saltzman Einhorn, Inc. www.eachmindmatters.org	The website hosts the CalMHSA-funded documentary “A New State of Mind: Ending the Stigma of Mental Illness” and a gallery of short videos telling personal stories of hope, resilience, and recovery. The site is also a hub for distributing CalMHSA resources across the SDR, suicide prevention, and student mental health PEI initiatives.	General public, gatekeepers	June 8, 2013, to June 30, 2014
Runyon Saltzman Einhorn, Inc. speakourminds.org		Organizations seeking to implement speakers bureaus	June 8, 2013, to June 30, 2014
Runyon Saltzman Einhorn, Inc. us.reachout.com/forums us.reachout.com/buscaapoyo us.reachout.com/reachouthere	These discussion forums were designed as part of a mobilization marketing campaign to encourage youth ages 14 to 24 to seek and provide peer support for emotional issues. ReachOut.com forums provide a moderated online discussion environment for teens and young adults to give and get peer support for emotional, relationship, and health issues. The forums are also available in a Spanish-language translation and adaption, BuscaApoyo. The ReachOutHere page provides an entry point to the forums and includes factsheets, stories, and information about how to get help for oneself or a friend. The web analytic data reported for the ReachOut.com forums include data from us.reachout.com/ReachOutHere, an informational access point for the ReachOut forums, and us.reachout.com/BuscaApoyo, the Spanish-language forums.	Youth ages 14 to 24	June 8, 2013, to June 30, 2014
Runyon Saltzman Einhorn, Inc. walkinourshoes.org	The website is an online companion to a school-based theater presentation targeting youth ages 9 to 13 to correct misperceptions about mental illness. WalkInOurShoes.org contains a video story gallery for adolescents to learn about experiences of mental health and recovery from other transition-age youth and add their own “shoe” to “stomp out stigma” to a gallery. There is additional supporting material for educators and parents.	Youth ages 9 to 13, educators, parents	August 19, 2013, to June 30, 2014
Runyon Saltzman Einhorn, Inc. ponteenmiszapatos.org	PonteEnMisZapatos.org is the Spanish-language translation and adaptation of walkinourshoes.org.	Youth ages 9 to 13, educators, parents	September 9, 2013, to June 30, 2014
United Advocates for Children and Families www.uacf4hope.org	The website for United Advocates for Children and Families contains a variety of resources for families and children with mental health challenges. These include pages on special education and for various target populations, such as military families and transition-age youth, as well as pages focused on mental health disorders common among children and adolescents. A page focused on SDR contains resources such as train-the-trainer curricula and a guide to holding community network roundtables.	General public, families and children with mental health challenges	June 8, 2013, to June 30, 2014

NOTE: All available web analytic data gathered prior to June 8, 2013, were previously reported in Burnam et al., 2014.

Website Evaluation Methods

For consistency of metrics and tracking, we opted to use the web analytics service provided by Google Analytics for all program partners creating online resources, rather than the various analytics services that program partners may have already implemented for their own tracking purposes. Google Analytics is the most widely used website statistics service; it is a free web analytics application that generates detailed statistics and information about website performance. We captured a wide range of metrics using Google Analytics to examine the use of and interaction with each website. Measures of traffic (site visits and page views) give a sense of the volume of visitors to the site. Measures indicating how users access the website (directly, referred from another website, or through search engines) can provide insight into the factors that motivated a user to visit the site; for example, a user who accesses a site directly may have been exposed to a campaign advertising a site name, whereas a user who visits through a search engine may have been looking for information on a topic related to what is covered on the site. Indicators such as average visit duration, average number of page views per visit, how many visits comprise only one page view (i.e., the bounce rate), and what topics or pages were most viewed give a sense of how users interacted with the site and for how long. Other useful metrics include those indicators that provide insight into where website visitors are located geographically, something of particular interest given the focus of CalMHSA's efforts on Californians. We excluded traffic from RAND during the monitoring period.*

There are some limitations to Google Analytics. The application does not include some relevant downloads (for example, those accessed from a link on another site or in an email) and so may underestimate them. Also, the location of the users' Internet connection approximates their actual geographic location, but users' networks and/or browsers can be configured to not share this information with analytics packages. For each website, there may be a small percentage of users for which we have no tracking data, so these users are excluded from analyses. Despite these limitations to describing user characteristics, website analytics can be used to benchmark website use, identify patterns of use, and describe the reach of the websites, which can ultimately inform how to best maintain, support, and adapt a website.

We note that the analytic data reported here pertain only to the CalMHSA-funded online resources hosted on the sites listed in Table 1. Many program partners engaged in online activities that extend beyond the resources listed here. For example, they may have provided materials to other organizations to host on their websites, provided input into the development of other organizations' online resources, or developed additional sites that were not tracked for this evaluation. In addition, this article does not track analytic data for other partner organizations that may have helped promote the online resources developed by CalMHSA program partners. As a result, the data reported here may underestimate the volume of online activity in which program partners are directly or indirectly engaged.

Overview of Results

Here, we provide an overview of the web metrics across all SDR program partners for which we collected web analytic data. We also note that we are not directly comparing metrics across the websites, because they vary widely in content and target audience, and some websites were monitored for different periods. We report tracking for most websites from June 8, 2013, to June 30, 2014; web analytic data for periods prior to this window, if available, were previously reported (Burnam et al., 2014).

Visits, Page Views, and Resources Downloaded

During the current tracking period (see Table 1 for tracking period dates for each site), 651,243 site visits were made to CalMHSA SDR websites, leading to 1,575,957 page views and 66,871 downloads of digital resources.** As expected—given that some websites target general audiences while others aim to reach more-specific audiences—numbers of visits ranged widely, from Disability Rights California's approximately 290,000 visits over 16 months to Runyon Saltzman Einhorn's SpeakOurMinds.org website's 3,600 visits over 13 months (see Table 2).

SDR Program Partner	Months of Tracking	Visits	Page Views	Downloads
Disability Rights California	16	292,832	580,945	45,739
Entertainment Industries Council	13	9,780	21,198	1,535
Integrated Behavioral Health Project/Center for Care Innovations	13	40,038	72,612	12,788
Mental Health Association of San Francisco	13	24,275	66,027	4,551
Runyon Saltzman Einhorn Each Mind Matters	13	57,328	116,449	538^
Runyon Saltzman Einhorn Speak Our Minds	13	3,633	11,009	953
Runyon Saltzman Einhorn ReachOut forums (includes Busca Apoyo and Reach Out Here)	13	127,074	469,235	Not available^^
Runyon Saltzman Einhorn Walk in Our Shoes	11	75,492	191,141	Not available^^
Runyon Saltzman Einhorn Ponte en Mis Zapatos	10	14,171	26,411	Not available^^
United Advocates for Children and Families	13	6,620	20,930	767
Total		651,243	1,575,957	66,871

^ Vimeo data of total number of downloaded videos. ^^ Downloads were not tracked for these sites.

User Engagement with Sites

We tracked the ways in which users arrived at or accessed the websites. Users could arrive from direct access (e.g., typing the URL into their browser, clicking on a bookmark, or following a link in an email), via a search engine such as Google or Yahoo!, or through referral via an external link on another website (e.g., clicking on a Facebook link to get to a site). Table 3 shows measures of how users engaged with the websites. For Entertainment Industries Council's TEAM Up site, Runyon Saltzman Einhorn's EachMindMatters.org, and Runyon Saltzman Einhorn's SpeakOurMinds.org, users accessed those websites in all three ways quite evenly. For example, Runyon Saltzman Einhorn's EachMindMatters.org had about one-third of users access the website in each way—direct, referral, and search engine. For three websites, Integrated Behavioral Health Project/Center for Care Innovations, United Advocates for Children and Families, and Runyon Saltzman Einhorn's ReachOut forums, the majority of users accessed those websites via search. In contrast with the other sites, the majority of users of the Runyon Saltzman Einhorn websites WalkInOurShoes.org (59 percent) and PonteEnMisZapatos.org (77 percent) accessed the websites from online advertising campaigns promoting the websites, which is a type of paid referral.

SDR Program Partner	Direct (%)	Referrals (%)	Searches (%)	Average time on site^	Average number of pages visited
Disability Rights California	39	9	50	3:35	2.14
Entertainment Industries Council	30	44	24	2:55	2.17
Integrated Behavioral Health Project/Center for Care Innovations	16	10	74	2:18	1.81
Mental Health Association of San Francisco	49	36	9	3:05	2.72
Runyon Saltzman Einhorn Each Mind Matters	36	25	39	2:19	2.03
Runyon Saltzman Einhorn Speak Our Minds	24	28	48	2:32	3.03
Runyon Saltzman Einhorn ReachOut forums (includes Busca Apoyo and Reach Out Here)	18	3	49	2:07	2.56
Runyon Saltzman Einhorn Walk in Our Shoes^^	18	5	12	2:04	2.53
Runyon Saltzman Einhorn Ponte en Mis Zapatos^^	7	3	13	1:38	1.86
United Advocates for Children and Families	25	17	57	2:58	3.16

NOTE: Direct, referral, and search percentages were calculated as the number of site visits of that type out of the total number of site visits. For Runyon Saltzman Einhorn Reach Out forums only, site visits are the number of site visits beginning at the forum, mode-specific and total. Percentages may not total 100 because less frequent modes such as email, mobile banner, campaign, and streaming were excluded from the table.

^ Time is displayed in minutes and seconds.

^^ For these Runyon Saltzman Einhorn websites, the Google Analytics settings were augmented to capture when users accessed the websites from online paid advertising campaigns.

We also tracked the average time that users spent on each site and the average number of pages visited while users were on the site. The average time spent on the site varied from about one and a half minutes (Runyon Saltzman Einhorn PonteEnMis-Zapatos.org) to nearly four minutes (Disability Rights California). The number of pages visited varied from about 1.8 (Integrated Behavioral Health Project/Center for Care Innovations and Runyon Saltzman Einhorn PonteEnMisZapatos.org) to 3.2 pages (United Advocates for Children and Families).

User Location

Because the CalMHSA effort is California-based and designed for reaching individuals and organizations across California, we examined the location of users accessing program partner websites (see Table 4). Google Analytics determines user location by the user's Internet Protocol (IP) address. This is a limitation of the metric because the geographic location indicated by the IP address may not always reflect the actual geographic location of the user; some reports indicate IP address location data is an accurate reflection of a user's actual geographic location to within a 25-mile radius (Clifton, 2012). Regardless, this metric can be used to observe general trends in user location. For six websites, the vast majority of users accessed the websites from California, ranging from 66 percent to 89 percent of users. For three websites, Entertainment Industries Council's TEAM Up site and Runyon Saltzman Einhorn's ReachOut forums and PonteEnMisZapatos.org, 35–47 percent of users were in California. Though users originated from many areas across California, across all websites, an average of 82 percent of the users originated from three of the largest metropolitan areas—the Los Angeles, Sacramento, and San Francisco areas.*** This is somewhat more than would be expected, given that 60 percent of Californians reside in these areas (U.S. Census, 2015).

SDR Program Partner	Visits from California	Visits from California (% of total visits)	Visits from the Los Angeles, Sacramento, and San Francisco metropolitan areas (% of California visits)
Disability Rights California	261,830	89	83
Entertainment Industries Council	4,024	41	87
Integrated Behavioral Health Project/Center for Care Innovations	7,230	18	80
Mental Health Association of San Francisco	16,888	70	89
Runyon Saltzman Einhorn Each Mind Matters	37,621	66	82
Runyon Saltzman Einhorn Speak Our Minds	2,384	66	77
Runyon Saltzman Einhorn ReachOut forums (includes Busca Apoyo and Reach Out Here)	45,064	35	79
Runyon Saltzman Einhorn Walk in Our Shoes	49,623	66	80
Runyon Saltzman Einhorn Ponte en Mis Zapatos	6,684	47	82
United Advocates for Children and Families	4,776	72	85
Total	436,124	67	82

NOTE: User characteristics, such as geographical location, determined by the user's IP address when visiting a website.

Discussion

RAND's evaluation of the utilization of the CalMHSA-funded SDR online resources shows that the websites have been visited many times—more than 650,000 visits in a year's time. Assuming the California-based visits were from unique individuals, this would indicate that, based on 2014 census estimates, 1.1 percent of California's population had visited the sites. However, because visits likely were not from unique individuals and we know that some visits originated from IP addresses outside the state, this represents an upper bound (and an overestimate) on the maximum potential reach of the online resources to Californians.**** We cannot determine how many individuals visited multiple CalMHSA-sponsored sites or how many represented previous visitors to these sites (in the case of sites that existed prior to receiving funding from CalMHSA).

Few studies assess the effects of public service announcements on driving traffic to affiliated websites, so it is unclear how big a change in web traffic should be expected for websites associated with a campaign (like Runyon Saltzman Einhorn's Each Mind Matters). One study showed that only 0.000061 percent of residents in states where an SDR campaign based on public service announcements was piloted visited an affiliated website (Corrigan, 2012); so the reach of CalMHSA sites to a very small percentage of California residents is not necessarily unexpected.

Nonetheless, the majority (60 percent) of visits to the websites originated from within California. Of those visits from within California, more than 80 percent originated from three of the largest metropolitan areas—the Los Angeles, Sacramento, and San Francisco areas. About 60 percent of Californians reside in these areas, suggesting that the reach of online resources was not as good to less populous areas of the state. This is desirable for a program like that of the Entertainment Industries Council, which targets entertainment professionals who mostly reside in these three metropolitan areas. For the overall SDR online resource effort, reach to less populous areas is one advantage of using online media, and this did not appear to be achieved.

Several analytic measures provide insight into how visitors to CalMHSA-funded SDR online resources engaged with the sites. Visitors to the sites downloaded more than 66,000 resources. Because the data do not allow us to determine how many resources each user downloads, 66,000 is an upper-bound estimate of the number of individuals who may have downloaded resources. The actual number of individuals who downloaded resources is likely less, assuming that at least some individuals downloaded multiple resources. Web analytic data indicate that, on average, users spent two to three minutes on the sites when they visited. Because Google Analytics' calculations of time on a site leave out the time spent on the last page visited, these estimates represent a conservative estimate of the time users spent on the sites.

Some program partners' efforts to reach Californians using online resources were particularly noteworthy in their success. For example, Disability Rights California's efforts to make antidiscrimination materials available yielded more than 292,000 visits, with more than half a million page views and more than 45,000 downloads of resource materials. Disability Rights California's website targeted general audiences, so broad reach is important in judging its success. In another example, Runyon Saltzman Einhorn's ReachOut forums, which targeted young adults between the ages of 14 and 24, resulted in more than 127,000 visits and more than 469,000 page views, suggesting that the forums were fairly heavily used. Other program partners' sites received less traffic, but were targeted at very specific, smaller groups. A good example is Entertainment Industries Council's TEAM Up site, which had less than 10,000 visits but was targeting media professionals. In another example, Runyon Saltzman Einhorn's Speak Our Minds site also targeted a specific audience—winners of grants for setting up speakers bureaus—and also had less than 10,000 visits. United Advocates for Children and Families had less than 10,000 site visits, even though that site was meant to be of fairly general interest.

These results suggest that future evaluation efforts should explore the strategies by which each program partner attempted to drive traffic to its site (e.g., through targeted advertising, advertising through partner websites). This type of exploration is beyond the scope of the current evaluation. Conducting such an evaluation would provide insight into whether reallocation of resources to improving dissemination efforts (e.g., better promotion of site presence, site reorganization to make available resources easier to find) or reallocation of funding to more-effective SDR efforts might be warranted.

Key Findings

• More than 650,000 visits were made to CalMHSA-funded SDR online resources.
• 436,000 visits originated in California, with over 80 percent from three of the largest metropolitan areas—the Los Angeles, Sacramento, and San Francisco areas.
• A maximum of 1.1 percent of California's population may have visited a CalMHSA-funded SDR online resource.
• Visitors downloaded more than 66,000 resources and spent an average of two to three minutes on the sites.

References

Burnam, M. Audrey, Sandra H. Berry, Jennifer L. Cerully, and Nicole K. Eberhart, Evaluation of the California Mental Health Services Authority's Prevention and Early Intervention Initiatives: Progress and Preliminary Findings, Santa Monica, Calif.: RAND Corporation, RR-438-CMHSA, 2014. As of July 29, 2015:
http://www.rand.org/pubs/research_reports/RR438.html

Clifton, Brian, Advanced Web Metrics with Google Analytics, 3rd ed., Indianapolis, Ind.: John Wiley & Sons, 2012.

Corrigan, P. W., “Where Is the Evidence Supporting Public Service Announcements to Eliminate Mental Illness Stigma?” Psychiatric Services, Vol. 63, No. 1, 2012, pp. 79–83.

Google, “Analytics Core Reporting API: Geo Network—Dimensions and Metro Explorer,” 2015. As of July 29, 2015:
https://developers.google.com/analytics/devguides/reporting/core/dimsmets#view=detail&group=geo_network&jump=ga_metro

TEAM Up, Social Media Guidelines for Mental Health Promotion and Suicide Prevention, Entertainment Industries Council, 2014. As of July 30, 2015:
http://www.eiconline.org/teamup/wp-content/files/teamup-mental-health-social-media-guidelines.pdf

U.S. Census Bureau, Population Division, “Annual Estimates of the Resident Population: April 1, 2010 to July 1, 2014,” 2015. As of July 29, 2015:
http://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?src=CF

U.S. Department of Health and Human Services, Mental Health: A Report of the Surgeon General, Rockville, Md., 1999.

Notes

* We were unable to exclude program partner traffic because there is no reliable way to determine exactly which traffic is from a program partner and which is not after data have been collected by Google Analytics. Using the service provider or network domain is a proxy for identifying which users are affiliated with a program partner, but may not include all program partner users.

** Download tracking was not available for all program partners' materials. Some program partners did not have CalMHSA-funded materials available for download. Some program partners made materials available during the evaluation period, but download tracking was not enabled for those items.

*** In Google Analytics, these areas are defined as “the designated market area (DMA) from where traffic arrived” (Google, 2015). Thus, these areas can include more than one city. The Sacramento area consists of areas in and around Sacramento, Stockton, and Modesto, and the San Francisco area consists of areas in and around San Francisco, Oakland, and San Jose.

**** There is no way to accurately assess the number of unique site visitors because HTTP cookies, the small pieces of data stored in users' browsers to “remember” interactions with websites and that are used by Google Analytics to measure how users interact with websites, are attached to individual browsers on computers, rather than people.

This research was conducted in RAND Health, a division of the RAND Corporation.