The National Science Digital Library/Distributed Learning (NSDL) program supported the development of resources, collections, and technical tools and services for teachers and learners in science, technology, engineering, and mathematics (STEM) education. This evaluation of the NSDL program assessed the sustainability of resources, tools, and services over time and across the changing technology landscape.

Digital Resources for STEM Educators and Recommendations for Cyberlearning Initiatives

Results from the National Science Foundation Digital Library/Distributed Learning Program Evaluation

Tora K. Bikson, Susan G. Straus, Grace Agnew, Nidhi Kalra, David J. McArthur, Helen Crompton, Courtney Ann Kase, Kristin J. Leuschner

ResearchPublished Apr 27, 2017

The National Science Foundation (NSF) created the National Science Digital Library/Distributed Learning (NSDL) program in 2000. Between 2000 and 2011, NSDL awarded more than 250 grants to support the development of resources, collections, and technical tools and services for teachers and learners in science, technology, engineering, and math (STEM) education. NSDL's mission was to assemble these elements into a distributed digital library that would be widely available to the public through websites. In 2011, NSF discontinued grant funding for projects in order to build on the results of the NSDL program and to establish the next generation of cyberlearning programs.

This report documents the third and final phase in a series of evaluations of the NSDL program. The broad goal of the evaluation was to assess the sustainability of NSDL resources, tools, and services over time and across the changing technology landscape. The report also identifies lessons that could be useful to other programs, both within and outside NSF — specifically, the attributes of sustainable digital initiatives and the extent to which the NSDL program demonstrated these attributes.

Key Takeaways

Audience

  • NSDL and the pathways have shown considerable improvement in identifying the audience served.
  • The NSDL portal shows strong audience engagement in terms providing capabilities for users to organize, comment on, contribute, and share resources.
  • NSDL exceled in development of paradata, which document resource use in the aggregate, and in the collection and harvesting of annotation metadata, which consist of users' comments and ratings for resources.

Organization

  • NSDL has a strong technical infrastructure, with a robust platform that is well suited to a metadata-based collection portal.
  • The NSDL portal and pathways have visionary leadership, but evidence was less clear for staff in operational roles to support a transition from research to sustainable business organizations.
  • Broad and diverse parent organizations and other stakeholders are a potential asset with respect to revenue for sustainability.

Collections

  • The NSDL portal and pathways showed improvements in the authoritativeness of their collections, professional oversight of resource accessions, and basic search services.
  • Accurate and complete metadata, alignment of educational standards to resources in metadata, and the expansion of usage metadata are significant contributions of the NSDL portal and some pathways, but few projects have shown how materials can be tailed to meet specific educational requirements or audiences.

Tools and Services

  • Funding supported the development of a diverse array of tools and services.
  • Although many NSDL projects created innovative products, there were redundant efforts across some projects, as well as gaps in the provision of needed tools and services.
  • The degree to which products were integrated with the NSDL portal was variable.

Recommendations

  • Improve audience identification by enabling users to customize resources to their workflows, providing a compelling rationale for users to select each portal over competing alternatives, providing stronger evidence of accountability to the audience, and broadening opportunities for audience engagement.
  • Improve organization by allowing for integrated security beyond basic user identification and password protection, providing support for users in terms of training and personalized services, and staffing roles to support the transition from research projects to nonprofit businesses.
  • Improve collections by publishing a sound copyright strategy, documenting copyright and permitted uses in resource metadata, and enabling searching from the front page of each site.
  • Improve tools and services by fostering collaboration among projects to avoid redundancy in development and fill gaps in services and ensuring that projects are integrated with the NSDL portal.
  • Future evaluation efforts of the health and sustainability of digital initiatives should use a broader range of methods and data sources, such as data from clients and customers and findings about impact on learning outcomes.
  • Cyberlearning initiatives, particularly multiyear, collaborative initiatives, need a much greater focus on sustainability from the outset to balance the emphasis that NSF has traditionally placed on innovation.
  • Cyberlearning initiatives should use a combination of initial grant funding, followed by contracts or cooperative agreements, to foster a balance between innovation and sustainability.
  • NSF should incorporate concrete sustainability requirements into solicitations and annual reporting requirements — in particular, to target funding directly to products that have been identified as sustainable and worthy of ongoing development and support.

Topics

Document Details

Citation

Chicago Manual of Style

Bikson, Tora K., Susan G. Straus, Grace Agnew, Nidhi Kalra, David J. McArthur, Helen Crompton, Courtney Ann Kase, and Kristin J. Leuschner, Digital Resources for STEM Educators and Recommendations for Cyberlearning Initiatives: Results from the National Science Foundation Digital Library/Distributed Learning Program Evaluation. Santa Monica, CA: RAND Corporation, 2017. https://www.rand.org/pubs/research_reports/RR414.html.
BibTeX RIS

The research reported here was conducted by the Science, Technology and Policy Program within RAND Justice, Infrastructure, and Environment.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.