The National Science Digital Library/Distributed Learning (NSDL) program supported the development of resources, collections, and technical tools and services for teachers and learners in science, technology, engineering, and mathematics (STEM) education. This evaluation of the NSDL program assessed the sustainability of resources, tools, and services over time and across the changing technology landscape.

Cover: Digital Resources for STEM Educators and Recommendations for Cyberlearning Initiatives
Read Online

Digital Resources for STEM Educators and Recommendations for Cyberlearning Initiatives

Results from the National Science Foundation Digital Library/Distributed Learning Program Evaluation

by Tora K. Bikson, Susan G. Straus, Grace Agnew, Nidhi Kalra, David J. McArthur, Helen Crompton, Courtney Ann Kase, Kristin J. Leuschner

Download eBook for Free

FormatFile SizeNotes
PDF file 3.6 MB

Use Adobe Acrobat Reader version 10 or higher for the best experience.

Research Questions

  1. What are the attributes of sustainable digital initiatives and to what extent did the NSDL program demonstrate these attributes?
  2. What lessons from the NSDL program could be useful to other programs, both within and outside NSF?

The National Science Foundation (NSF) created the National Science Digital Library/Distributed Learning (NSDL) program in 2000. Between 2000 and 2011, NSDL awarded more than 250 grants to support the development of resources, collections, and technical tools and services for teachers and learners in science, technology, engineering, and math (STEM) education. NSDL's mission was to assemble these elements into a distributed digital library that would be widely available to the public through websites. In 2011, NSF discontinued grant funding for projects in order to build on the results of the NSDL program and to establish the next generation of cyberlearning programs.

This report documents the third and final phase in a series of evaluations of the NSDL program. The broad goal of the evaluation was to assess the sustainability of NSDL resources, tools, and services over time and across the changing technology landscape. The report also identifies lessons that could be useful to other programs, both within and outside NSF — specifically, the attributes of sustainable digital initiatives and the extent to which the NSDL program demonstrated these attributes.

Key Findings

Audience

  • NSDL and the pathways have shown considerable improvement in identifying the audience served.
  • The NSDL portal shows strong audience engagement in terms providing capabilities for users to organize, comment on, contribute, and share resources.
  • NSDL exceled in development of paradata, which document resource use in the aggregate, and in the collection and harvesting of annotation metadata, which consist of users' comments and ratings for resources.

Organization

  • NSDL has a strong technical infrastructure, with a robust platform that is well suited to a metadata-based collection portal.
  • The NSDL portal and pathways have visionary leadership, but evidence was less clear for staff in operational roles to support a transition from research to sustainable business organizations.
  • Broad and diverse parent organizations and other stakeholders are a potential asset with respect to revenue for sustainability.

Collections

  • The NSDL portal and pathways showed improvements in the authoritativeness of their collections, professional oversight of resource accessions, and basic search services.
  • Accurate and complete metadata, alignment of educational standards to resources in metadata, and the expansion of usage metadata are significant contributions of the NSDL portal and some pathways, but few projects have shown how materials can be tailed to meet specific educational requirements or audiences.

Tools and Services

  • Funding supported the development of a diverse array of tools and services.
  • Although many NSDL projects created innovative products, there were redundant efforts across some projects, as well as gaps in the provision of needed tools and services.
  • The degree to which products were integrated with the NSDL portal was variable.

Recommendations

  • Improve audience identification by enabling users to customize resources to their workflows, providing a compelling rationale for users to select each portal over competing alternatives, providing stronger evidence of accountability to the audience, and broadening opportunities for audience engagement.
  • Improve organization by allowing for integrated security beyond basic user identification and password protection, providing support for users in terms of training and personalized services, and staffing roles to support the transition from research projects to nonprofit businesses.
  • Improve collections by publishing a sound copyright strategy, documenting copyright and permitted uses in resource metadata, and enabling searching from the front page of each site.
  • Improve tools and services by fostering collaboration among projects to avoid redundancy in development and fill gaps in services and ensuring that projects are integrated with the NSDL portal.
  • Future evaluation efforts of the health and sustainability of digital initiatives should use a broader range of methods and data sources, such as data from clients and customers and findings about impact on learning outcomes.
  • Cyberlearning initiatives, particularly multiyear, collaborative initiatives, need a much greater focus on sustainability from the outset to balance the emphasis that NSF has traditionally placed on innovation.
  • Cyberlearning initiatives should use a combination of initial grant funding, followed by contracts or cooperative agreements, to foster a balance between innovation and sustainability.
  • NSF should incorporate concrete sustainability requirements into solicitations and annual reporting requirements — in particular, to target funding directly to products that have been identified as sustainable and worthy of ongoing development and support.

Table of Contents

  • Chapter One

    Introduction

  • Chapter Two

    Overview of the NSDL Program

  • Chapter Three

    Sustainability of Digital Collection Initiatives: A Literature Review

  • Chapter Four

    Health and Sustainability of Collected Resources

  • Chapter Five

    Digital Tools and Services

  • Chapter Six

    Usability of NSDL for Lesson Planning

  • Chapter Seven

    Full Instantiation of the NSDL Logic Model: Replicated Case Studies

  • Chapter Eight

    Conclusions and Recommendations

  • Appendix A

    NSDL Organization Leadership

  • Appendix B

    Evaluation of the NSDL Technical Platform

  • Appendix C

    Key NSDL Tools and Services

  • Appendix D

    CPALMS and iCPALMS Funding Sources

  • Appendix E

    Proposed Teachers' Survey

The research reported here was conducted by the Science, Technology and Policy Program within RAND Justice, Infrastructure, and Environment.

This report is part of the RAND Corporation Research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.