Program Evaluation

“The child shall enjoy special protection, and shall be given opportunities and facilities, by law and by other means, to enable him to develop physically, mentally, morally, spiritually and socially in a healthy and normal manner and in conditions of freedom and dignity” (Principle 2, Declaration of the Rights of the Child 1959).

This declaration prompted the designation of an annual Children's Day, celebrated in many European countries on the first day of June. On this date, children typically receive presents from their parents and participate in various sport and cultural activities. However, the historic objective of Children's Day was not simply to celebrate children for who they are, but to bring attention to children around the world who suffer from exploitation, violence and discrimination. Raising awareness and strengthening children's rights have long been objectives for many international and national organizations.

This commentary appeared on Spotlight on Poverty and Opportunity on May 14, 2013.

At a time when government finances are stretched, there is growing interest in finding new and innovative ways to privately fund the public services that improve social outcomes. One approach that has garnered significant interest among policymakers and advocates is the Social Impact Bond, or SIB. Early experiments in the United Kingdom are testing whether these bonds could be part of the solution to reducing pressure on public spending, but, to help SIBs fulfil this promise, it's critical to address how they can be most effectively implemented.

Originating in the United Kingdom, SIBs were first brought to the United States in August 2012 when the Goldman Sachs Group committed nearly $10 million to an anti-recidivism program for adolescents at Rikers Island prison in New York.

Drug and alcohol use among the nation's youth remains a problem. While evidence-based prevention programs are available, practitioners, especially those in smaller, community-based settings, often have difficulty using them. This is because resources are limited, prevention is complex, and communities often lack the capacity to adapt and implement “off the shelf” programs. A result of this “gap” between research and practice is that communities and schools often do not use the most evidence-supported programs, and often implement them poorly when they do.

For example, a recent report (PDF) by the Department of Education showed that fewer than 10 percent of schools were implementing evidence-based drug prevention programs, and fewer than half of those schools were doing a good job of implementing those programs. Common ways to bridge this gap, such as information dissemination, fail to change practice or outcomes at the local level, in part because they do not sufficiently address capacity or use community input. Building a community's prevention capacity through greater collaboration between researchers and practitioners could improve the quality of prevention and outcomes.

This commentary appeared in Research Europe on April 25, 2013.

Awarding money for research is not easy. As physicist Niels Bohr once remarked: “Prediction is hard, especially about the future.” Funders are expected to judge which research proposals will turn out best even though how they will turn out is, by definition, unknown.

The traditional way of doing this is peer review. In the UK, more than 95 per cent of grant money for biomedical research is allocated in this way....

The remainder of this commentary is available at researchresearch.com.

Much of today's research on health problems tests new treatments under ideal conditions. However, providers, especially those in smaller community-based settings, often have difficulty using these new treatments. This is because community-based providers operate under much tougher conditions than the conditions in which the treatments were tested. They usually have less funding, equipment, expertise, time, and staff—collectively called capacity—than the original research trials. Thus, if the first problem of health interventions is finding new and better treatments, getting those treatments to patients across the range of health delivery settings can be thought of as the “second problem.”

A new field called implementation science has been developed in the last 5 to 10 years to specifically address this “second problem”—that is, how to best support providers to take up new, research-proven treatments and implement them well....

This commentary appeared on Independent Sector on March 21, 2012.

One of my interests is building evaluation capacity within nonprofit community service organizations to increase impact. The model RAND uses for this work is called Getting To Outcomes (GTO), a free program management tool with demonstrated success in improving program performance.

Getting To Outcomes emphasizes Continuous Quality Improvement (CQI), which we define as a planned, structured process with the goal of improving outcomes. We have adapted the Plan-Do-Study-Act (PDSA) cycle approach for use in community service settings, where data and resources to conduct analyses are often limited as compared with settings where PDSA was first developed (e.g., manufacturing) and currently used (e.g., hospitals). Plan-Do-Study-Act allows organizations to pilot small changes and see if these changes enhance service delivery and outcomes. Inherent in this approach is the use of data to analyze a program or process and understand the impact of applying the PDSA cycle. Critical to PDSA are support from leadership and the inclusion of multiple levels of the organization (i.e., a team-based strategy) to plan, design, implement, and study impact. For example, we've found that it is critical for leaders to support a dialogue among program staff about the potential strengths and weaknesses of a particular program and to support and provide resources to staff to make changes. Another important component of CQI is that it is ongoing. That means that it is not a one-time event, but should be incorporated into regular staff meetings for progress monitoring and mid-course adjustments, as well as for celebrating successes and problem solving setbacks.

Increasingly governments in the UK and throughout Europe see R&D driven growth as the best way out of the financial crisis. Western economies, the argument goes, cannot compete with the low labour costs of manufacture base in China and India. Since the boom of the financial sector went bust, the best of what is left are our knowledge, ideas and our capacity to innovate.

The Government identifies the "UK's world-class research base" as a key driver of economic growth. But does the evidence of returns from public investment in R&D match the rhetoric?