Evaluation of CalMHSA Student Mental Health Online Resources
RAND Health Quarterly, 2016; 6(1):10
RAND Health Quarterly, 2016; 6(1):10
RAND Health Quarterly is an online-only journal dedicated to showcasing the breadth of health research and policy analysis conducted RAND-wide.
More in this issueAnalysis of visitors to CalMHSA-funded student mental health websites shows that visitors at educational institutions comprised the largest audience segment and promotional campaigns likely increased traffic.
Mental health disorders among children and adolescents have received increased attention in the aftermath of the Surgeon General's warning in 2000 that the United States is facing a “public crisis in mental healthcare for infants, children and adolescents” (U.S. Department of Health and Human Services, U.S. Department of Education, and U.S. Department of Justice, 2000). Schools play a key role in providing mental health services to youth. Primary and secondary school–age students are more likely to receive mental health services at school than in any other setting (Burns et al., 1995; Jaycox et al., 2010). School and campus mental health services, staff, and faculty are uniquely positioned to recognize students at risk for mental health problems and to respond when problems arise. Moreover, school faculty and staff play an important role in mental health prevention and early intervention (PEI) strategies by educating students and parents, identifying student at-risk behaviors, and referring students for appropriate mental health services.
Improving student mental health in California's public K–12 and higher-education systems is one of the primary objectives of the California Mental Health Services Authority (CalMHSA). CalMHSA's statewide student mental health initiative, part of the PEI program that was funded by the Mental Health Services Act (Proposition 63) of 2004, includes a range of activities and tools to promote and protect student mental health and wellness. Online informational resources are a key component of this initiative. With CalMHSA's support, the California County Superintendents Educational Services Association (CCSESA)—an organization that helps to design and implement statewide K–12 programs that are cost-effective, coordinated, and efficacious for students, schools, and regions—the 10-campus University of California (UC) system, the 23-campus California State University (CSU) system, and the 112-campus California Community Colleges (CCC) system all have developed websites to raise awareness of student mental health issues and to inform faculty, staff, and higher-education students about options and best practices for addressing mental health issues. These websites have the potential to indirectly impact nearly 6.2 million students in California's public K–12 system and nearly 2.1 million students in California's public higher-education system.
As part of an ongoing evaluation of CalMHSA's statewide PEI initiatives, analysts with the RAND Corporation have collected and analyzed data on the use of online resources designed to improve student mental health in California's public K–12 and higher-education systems. This article provides a high-level overview of the evaluation to date of the content and utilization of the CCSESA, UC, CSU, and CCC student mental health websites. Our goal was to assess the potential reach of these websites and inform future website dissemination efforts. To this end, we examined how often users access the websites; how users got to the websites; which functions on the individual websites got the most use; and user characteristics, such as geographic location. In addition, we examined whether patterns of use change over time.
Using Google Analytics—the industry standard application for web analytics (W3Techs, undated)—we captured a wide range of metrics on the use of and interaction with the CCSESA, UC, CSU, and CCC websites, as well as traffic sources and additional information on user characteristics. Google Analytics generates statistics and information about website performance; measures include indicators of user traffic to the websites (e.g., the number of visits and page views), how users engage with and navigate the websites (e.g., the average number of page views per visit), and how users access the websites (e.g., typing the URL directly in a web browser, being referred to the URL from another website, or through keyword searches on search engines). Other data from Google Analytics include user characteristics, such as geographic location and type of Internet service provider (ISP) being used to access the website (e.g., university/college, school district, or county); when available, these data can help describe reach and identify patterns of use. For a complete list of terms and definitions, see the glossary.
CCSESA and the UC and CCC systems developed and launched their websites between 2012 and 2013, allowing for multiple years of data collection. CSU launched their website later, in 2014; as a result, our data collection period for the CSU website was limited. For this article, we examined all website utilization data available, ranging from the 12-month duration of data collection for the CSU website to the 30-month duration of data collection for the CCSESA website.
In this section, we provide brief descriptions of the CCSESA, UC, CSU, and CCC student mental health websites and their usage metrics. In addition to describing the educational context, target audience, and functionality of each website, we highlight our Google Analytics findings, with a specific focus on the traffic metrics and user engagement data that are most germane to each site.
As stated above, CCSESA supports county superintendents in the design and implementation of statewide K–12 education programs. In collaboration with the Sacramento County Office of Education and Regional Lead County Offices of Education, CCSESA developed an interactive website that serves as a clearinghouse of resources and information on K–12 student mental health issues. The target audience of the CCSESA website is preschool and K–12 teachers, school staff, and administrators, as well as mental health providers, parents, caregivers, and community members who work within preschool and K–12 school settings. The website includes links to school-based PEI program websites, downloadable files (e.g., articles, reports, summaries, fact sheets, bulletins, newsletters, and brochures related to student mental health), and descriptions of regional activities that promote student mental health and wellness.
Highlighted programs and practices are rated, using clearly defined criteria (i.e., evidence-based practice, promising practice, or emerging practice). The website is organized around 14 targeted resource topics, ranging from “anger management,” to “parent/family/community collaboration,” to “youth development.” Each resource topic page offers links to three sections—Publications & Tools, Programs & Practices, and Implementation. Finally, the CCSESA site includes a search function that allows users to search by keyword, resource type, topic, target audience, program rating level, grade level, format, and Response to Intervention (RTI) level.
Traffic and navigation. During the 30 months of tracked activity on the CCSESA website, there were 21,603 visits (on average, 720 visits per month) and 65,855 page views (on average, 2,195 page views per month). Users averaged 3.04 page views per visit. Based on data from September 2012 to February 2015, the website experienced several peaks in traffic: March 2013, October 2013, March 2014, June 2014, and September to October 2014 (see Figure 1). Although we were unable to identify activities that contributed directly to these peaks in website usage, these spikes were likely the result of email blasts and website awareness campaigns conducted during these months.
User access. Figure 2 illustrates the different ways that people accessed the CCSESA website during the 30-month reporting period. Direct access was the most common method of connecting to the website during the reporting period, with nearly one-half of the users (49 percent) accessing the CCSESA website either by typing the URL into a browser, clicking on a bookmark, or following a link in an email or other electronic document such as a PDF. The remaining visits to the website were by way of a search engine (27 percent), such as Google or Yahoo, where users entered a search term that returned a link to CCSESA's website, or via external links from other websites (24 percent). These data suggest that the majority of people who visited the CCSESA website either had direct knowledge of the site or encountered links to the site as a result of promotional activities.
When we analyzed traffic sources by quarter (3-month periods), we found an increase in access to the website via search engines and external links and a decrease in direct access (see Figure 3). During the first quarter of the reporting period, the proportion of direct access users was over 70 percent, but direct access decreased over time to less than 30 percent during the last quarter. The increase in search and referral traffic and the decline in direct traffic are to be expected as awareness of the website increases among key stakeholders.
To better understand factors related to the increase in traffic to the CCSESA website via search engines, we examined the rate at which search terms suggestive of awareness of the website's content—for example, “clearinghouse,” “student mental health initiative [SMHI],” or “regional K12”—were used during two different reporting periods. We compared targeted searches, which are distinguished by specific search terms, to random searches, which are distinguished by generic search terms. Comparing the October–December 2012 reporting period to the January–March 2014 reporting period, we found an increase of 188 percent in the use of specific search terms. Additionally, during this period, we saw a 1700-percent (or 18-fold) increase in the use of terms related to resource topics on CCSESA's website (for example, “anger management,” “behavior management,” or “bullying”). The finding that a large portion of the terms were specific, and that there was a substantial shift from random to targeted searches, implies that people were aware of the CCSESA website and were looking for specific information about K–12 student mental health issues.
As noted previously, CCSESA's website offers information and downloadable materials for 14 targeted resource topics. We found that users most often viewed the “anger management” topic, with 5,874 page views (33 percent of resource section page views); followed by “mental health/wellness” (2,496 page views; 14 percent) and “bullying” (2,042 page views; 11 percent). This finding may be influenced, however, by the fact that “anger management” is the first topic listed. Looking into how users navigated the different sections within each resource topic, we found that users were most likely to access resources in Programs & Practices (56 percent), which describes and contains recommendations for effective prevention and early identification strategies, models, and activities that have been implemented at the local level. After Programs & Practices, users were most likely to access resources in Publications & Tools (42 percent), which includes links to articles, reports, summaries, fact sheets, bulletins, newsletters, brochures, and websites containing evidence-based research and other helpful information to promote student mental health and wellness, and Implementation (3 percent), which includes information and tools to support effective implementation of prevention and early identification programs and practices. (Percentages rounded to the nearest percent.)
The 10-campus University of California system has a combined student body of approximately 240,000 students. The UC Student Mental Health and Promising Practices website is a comprehensive resource repository for UC system–wide sharing of data and information on student mental health issues. Hosted by the UC Office of the President, the website showcases the UC system's collective efforts to meet the mental health needs of UC students, staff, and faculty. The website is designed to reach a broad audience, including staff, faculty, students, and mental health providers from the different UC campuses and from surrounding county and community agencies. The homepage includes contact information and links to UC campus counseling centers, as well as links to information on suicide prevention, assisting students in distress, PEI training and programs, and conference information.
Traffic and navigation. During the 17 months of tracked activity on the UC student mental health site, there were 5,333 visits (on average, 314 visits per month) and 17,185 page views (on average, 1,011 page views per month). Users averaged 3.22 page views per visit. As illustrated in Figure 4, there was a notable peak in traffic (visits and page views) in November 2013, followed by a decline through February 2014. Then, in June and September 2014, there were two additional yet smaller peaks in page views, but these peaks were followed by another decline in traffic.
User access. Nearly one-half of users (48 percent) accessed the website directly (see Figure 5) and slightly fewer (43 percent) accessed the website from a search engine. During the reporting period, few users accessed the site by way of an external link on another website (9 percent).
The methods by which users accessed the UC student mental health website shifted over the course of the 17 months (see Figure 6). From October 2013–June 2014, the majority of users (50 percent to 80 percent) accessed the website directly. However, from July 2014–February 2015, the majority of users accessed the website by way of a search engine (60 percent to 70 percent). In October 2013, the UC website experienced high rates of direct traffic, which is in line with promotional activities that occurred at this time: UC officials sent out emails and a newsletter to campuses and program partners promoting the launch of the website. In the following two reporting periods, the use of search engines to access the website increased steadily from 5 percent (October–December 2013) to over 40 percent (April–June 2014), suggesting an increase in the use of search terms related to the website.
CSU's Mental Health Advocacy and Promotion (MHAP) website provides information and resources pertaining to the mental health of CSU students—of which there are roughly 444,000 enrolled at 23 campuses—as well as college students and young adults in general. The site is geared toward higher-education students and personnel, including educators and mental health professionals, but it also invites use by the general public; consumers of mental health services; and those who advocate for, promote, and support mental health and well-being. The CSU website was designed to engage users across all higher-education institutions and communities, not just those within California. The website connects users to information about PEI programs; treatment programs; education and professional development options; resources for faculty, staff, and parents; and programs and best practices. CSU's MHAP website is hosted within CSU's MERLOT (Multimedia Educational Resource for Learning and Online Teaching) system, a repository of information for faculty, staff, and students across institutions of higher education.
Traffic and navigation. During the 12 months of tracked activity on the CSU mental health website (March 2014–February 2015), there were 2,625 visits (on average, 219 visits per month) and 4,605 page views (on average, 384 page views per month). Users averaged 1.75 page views per visit. As illustrated in Figure 7, there were spikes in traffic (in both visits and page views) in July 2014, September 2014, and November 2014; however, these peaks were most pronounced for page views, while visits exhibited slow but steady growth over the tracking period.
User access. The majority of users (67 percent) accessed the CSU website by way of search engines throughout the reporting period (see Figure 8). However, because 89 percent of the search terms were not available (Google Analytics did not capture them because of user privacy settings), we were unable to determine whether particular themes for the search terms emerged. Twenty-three percent of users accessed the website directly; the remaining users accessed the website via external links from another website (10 percent). Of the 254 referral visits, 43 percent originated from the counseling and psychological services website of Humboldt State University, where CSU's student mental health site is recommended as a place to “find great information and resources.”
Methods by which users accessed the CSU website shifted slightly over the course of the reporting period (Figure 9). Over time, direct access decreased somewhat, whereas access via search engines increased. Despite these shifts, search engines remained the most common way for users to access the CSU website.
The California Community Colleges serve over two million students on 112 campuses. The CCC website functions as a portal for training and technical assistance for CCC's Student Mental Health Program (SMHP) and was developed and managed through a contract with the Center for Applied Research Solutions. The target audience of this website is CCC staff, faculty, mental health providers, professionals, and community partners (e.g., county mental health and health care providers, parent advocates, other higher-education systems). The homepage includes a program overview; links for training and regional events, toolkits, and webinars (with video access to archived webinars); and downloadable brochures and slideshows. The website also provides access to CalMHSA program resources, information on funding, and information on program evaluation. Resources available include downloadable web banners, radio spots (mp3 files) for promoting programs, training materials, program guides, campus SMHP profiles, and detailed program evaluation information.
Traffic and navigation. Over the course of the 23 months of tracked activity on the CCC SMHP website, there were 35,212 visits (on average, 1,531 visits per month) and a total of 79,890 page views (on average, 3,473 page views per month). Users averaged 2.27 page views per visit. Looking at traffic patterns across 23 months, we noticed two distinct peaks in traffic: a tremendous surge in visits occurred between October–December 2013, jumping from 3,781 page views in September of that year to 12,900 page views in November (see Figure 10).
In late November and early December 2013, the CCC system conducted a large email campaign to increase awareness of the website's newly launched, searchable student mental health database. During this period, we saw substantial growth in direct visits (from 1,036 to 10,405). Additionally, we found that direct visits during this quarter came from various metropolitan regions of California (e.g., 41 percent from Los Angeles, 14 percent from San Diego, 14 percent from San Francisco-Oakland-San Jose, 13 percent from Sacramento-Stockton-Modesto, and 8 percent from Fresno) and from a variety of network domains and service providers (e.g., over 50 percent from commercial ISPs such as AT&T and Comcast, 25 percent from university or college ISPs), rather than from any one distinct group. This finding suggests that the campaign to promote the website was successful not only in generating more visits to the site, but also in attracting a broader audience from across California.
In addition to a surge in direct visits during this quarter, there was also a spike in referral visits (from 384 in the prior quarter to 3,885). In November 2013, the CCC campuses launched a CalMHSA student mental health campus-wide survey for students, faculty, and staff—this online survey was hosted by RAND. At the close of the survey, all users were redirected to the CCC SMHP website, accounting for this peak in referral visits.
A second peak in overall activity, about one-half the size of the first, occurred during November 2014, with 4,947 visits. In this reporting period, we observed an increase in referred visits from the CCC Chancellor's Office website. Likely accounting for some of this peak in activity, CCC launched the second wave of the CalMHSA campus-wide survey in this reporting period, again redirecting respondents to the CCC SMHP website at the end of the survey. Although the UC and CSU systems also participated in the CalMHSA campus-wide survey, only the CCC system elected to redirect respondents to its student mental health website; UC and CSU respondents were redirected to the RAND Corporation website (www.rand.org).
User access. The majority of users (64 percent) accessed the CCC website directly over the reporting period (see Figure 11). The remainder of users accessed the site by way of an external link on another website (21 percent) or from a search engine (15 percent). As was the case with CCSESA, these findings suggest that most website traffic was a result of efforts to encourage awareness of and engagement with the CCC website through email campaigns or other promotional efforts. The ways in which users accessed the website remained consistent throughout the reporting period; thus, there were no changes in the ways users accessed the website to display in the report.
One of the features particular to the CCC SMHP website is a searchable database that allows users to search for programs, policies, and practices that promote student mental health across the community college system. Users can either enter a keyword into an open search box or search mental health resources by one or more sets of criteria presented on the website (e.g., topic, population of interest, intended audience). From November 2013–February 2015, users conducted a total of 7,182 searches and 6,187 unique searches (multiple searches by the same visitor in the same visit count as one unique search). Of the 7,182 total searches, there were only 555 (8 percent) keyword searches. Among these keyword searches, we observed no obvious themes or ways to group the searches, suggesting that the key terms used were quite diverse. In contrast, there were 3,839 (53 percent of total searches) criteria searches (e.g., a user clicked on a checkbox to select a specific topic, population, or intended audience) and 2,703 (38 percent of total searches) “view all resources” searches, suggesting that users may have used the database to browse resources available on CCC campuses rather than to search for a particular topic or issue. The most commonly searched criteria included “threat assessment/crisis” for topic area, “first-generation students” for population, and “students” for intended audience. Table 1 outlines the top three criteria searches by category. These top searches were consistent across the reporting period.
Category | Criteria | Times Searched |
---|---|---|
Topic | 1. Threat assessment/Crisis | 137 |
2. Behavioral intervention/Assessment teams | 121 | |
3. Mental Health 101 | 85 | |
Population | 1. First-generation students | 80 |
2. LGBTQ | 77 | |
3. General student population | 64 | |
Intended audience | 1. Students | 63 |
2. Faculty | 55 | |
3. Administration | 52 |
Note: Data collected from 11/01/2013 to 2/28/2015.
Our evaluation had several limitations. Because users' networks and/or browsers can be configured to not share information with Google Analytics and other analytics packages, the data presented in this article may underestimate the reach of the websites to some of the target audiences. However, the withholding of information or a user's privacy settings did not have an impact on our key traffic metrics; we have counts of all visits and page views made by users. While we do not have the geographic locations of all users when they visited the website, the frequency of missing information is small; for instance, fewer than 1 percent of users of the CCC website did not provide information on their location.
Although Google Analytics allowed us to capture a range of website metrics that illuminate patterns of use, how users access websites, and some user characteristics, we were unable to evaluate either the perceived utility and quality of the websites from the perspective of the user. Both of these factors are important in understanding the value-added of the websites to the current resources related to student mental health. Moreover, defining desirable outcomes for the websites varies widely depending on the website design, the expected size of the audience, and the breadth of the content.
We are also limited in our ability to make direct comparisons across websites due to the challenge of defining the target audiences. Many of the CalMHSA-funded student mental health websites contain resources that could be of use beyond the different campuses or education systems. For example, CCSESA's website provides a clearinghouse of resources and regional best practices not only for Pre-K through 12th-grade teachers, staff, or administrators, but also for mental health providers, parents, caregivers, and community members who partner in some way with schools across California. CSU's website broadly targets faculty, staff, and students of higher education, consumers of mental health services, and advocates of mental health issues both within and outside California.
Finally, we were unable to analyze whether users engaged with the websites as originally intended. Many of the sites underwent multiple revisions, improving the design and user engagement experience. Comparing the end result to the original proposed website plan has the potential to misrepresent quality improvement efforts. Despite these limitations, the metrics provided in this article can be used to describe the reach of the websites and to identify patterns of use, both of which can ultimately inform how to best maintain, support, and adapt a student mental health website.
Our evaluation of the CCSESA, UC, CSU, and CCC student mental health websites demonstrated that these CalMHSA-funded online resources served a large number of users, with a combined total of 64,973 visits (average monthly rates ranging from 238 to 1,531 visits) and 167,535 total page views. Most visitors (49 percent to 63 percent) to the CCSESA, UC, and CCC websites accessed the websites directly, by typing the specific URL into their browser, clicking on a bookmark, or following a link in an email or other electronic document such as a PDF. These findings suggest that email and other awareness campaigns succeeded in promoting early engagement with these websites, and reinforce the value of promotional activities that disseminate information about the websites.
For the higher-education systems, more than three-quarters (78 percent) of visits to the websites originated from within California. Although this indicates that a large proportion of users were based in California (a key characteristic of the intended audience of each of the CalMHSA-funded websites), the overall number of visits represents only a small percentage of the target population of all higher-education students, staff, and faculty in California. Most visitors to the CSU website accessed the site by way of a search engine and originated from outside California (61 percent). Because the CSU website targets users across all higher-education institutions, the lower percentage of users from California and the high use of search engines to access the website suggest that CSU succeeded in engaging a national audience, not just a California one. We anticipated that the websites affiliated with the higher-education systems (UC, CSU, and CCC) would draw heavily from users associated with universities or colleges, whereas the K–12 website (CCSESA) would draw from users associated with local or state-level K–12 entities. Data from ISPs support this assumption: a large percentage of visitors (22 percent to 44 percent) to the UC, CSU, and CCC student mental health websites were associated with college or university ISPs, whereas the largest category of visitors to the CCSESA website was associated with County Office/Department of Education ISPs (19 percent). These data suggest that a portion of the users accessing the CCSESA, UC, CSU, and CCC student mental health websites were from the intended target audience of the CalMHSA-funded websites; however, information about a large proportion of users still remains unknown.
The increase in the proportion of visits to the websites by way of search engines suggests a growing awareness of the websites and thus greater use of search terms related to the sites' content. Analysis of the peaks in usage also suggests that promotional campaigns have successfully increased website traffic immediately following their release and dissemination. Based on these findings, perhaps one goal for these websites is to use short, intensive outreach efforts to get on the map as an information source for student mental health issues, and then remain available on an as-needed basis for the long term.
Finally, based on our analyses of resource topics and search criteria on CCSESA and CCC's websites, there is a high level of interest in gathering information about evidence-based and promising practices for K–12 and higher-education PEI programs. Our findings suggest that users may be interested in browsing available databases and resources for information on student mental health issues and programs in general, rather than searching for information on a particular topic or resource.
Because the CCSESA, UC, CSU, and CCC student mental health websites are relatively inexpensive to maintain, and because our evaluation suggests an increasing use of these websites, we recommend efforts to sustain and publicize these web-based resources, as well as efforts to “cross-fertilize” ideas across websites, thus encouraging the use of material and approaches that are drawing the most views.
Online Resource | Website Description | Target Audience | Website Launch | Tracking Dates |
---|---|---|---|---|
California County Superintendents Educational Services Association (CCSESA) http://www.regionalk12smhi.org/ |
Online clearinghouse of best, promising, and community-defined practices (with a focus on grades K–8) | Pre-K–12 teachers, staff, or administrators; mental health providers, parents, caregivers, and community members who work with schools within California | 9/10/2012 | 9/2012 to 2/2015 (30 months) |
University of California (UC) http://www.ucop.edu/student-mental-health-resources/index.html |
Web-based resource repository (“clearinghouse”) for system-wide sharing of data and information on student mental health issues | UC faculty, staff, and students; and mental health providers from the various college/university, county, and community agencies within California | 10/24/2013 | 10/2013 to 2/2015 (17 months) |
California State University (CSU) http://teachingcommons.cdl.edu/=mhap/index.html |
Web-based repository of information for faculty, staff, and students across California-based and national institutions of higher education | Faculty, staff, and students of higher education; consumers of mental health services; and advocates of mental health issues within and outside California | 3/19/2014 | 3/2014 to 2/2015 (12 months) |
California Community Colleges (CCC) http://www.cccstudentmentalhealth.org/main.php |
Online repository of resources, materials, and policies developed under the CCC Student Mental Health Program (SMHP), funded by CalMHSA and in partnership with the Foundation for Community Colleges and Center for Applied Research and Solutions | CCC faculty, staff, mental health providers, professionals, and partners within California | 6/04/2012 | 4/2013 to 2/2015 (23 months) |
Burns, B. J., E. J. Costello, A. Angold, D. Tweed, D. Stangl, E. M. Z. Farmer, and A. Erkanli, “Children's Mental Health Service Use Across Service Sectors,” Health Affairs, Vol. 14, No. 3, 1995, pp. 147–159.
Jaycox, L. H., J. A. Cohen, A. P. Mannarino, D. W. Walker, A. K. Langley, M. Scott, and M. Schonlau, “Children's Access to Mental Health Care Following Hurricane Katrina: A Field Trial of Trauma-Focused Psychotherapies,” Journal of Traumatic Stress, Vol. 23, No. 2, 2010, pp. 223–231.
U.S. Department of Health and Human Services, U.S. Department of Education, and U.S. Department of Justice, “Conference Summary—Report of the Surgeon General's Conference on Children's Mental Health,” web page, U.S. Department of Health and Human Services, 2000. As of January 27, 2016:
http://www.ncbi.nlm.nih.gov/books/NBK44237/
W3Techs, “Usage of Traffic Analysis Tools for websites,” web page, undated. As of December 18, 2015:
www.w3techs.com/technologies/overview/traffic_analysis/all
This research was conducted in RAND Health, a division of the RAND Corporation.
RAND Health Quarterly is produced by the RAND Corporation. ISSN 2162-8254.
Explore RAND Health Quarterly articles on PubMed