This report outlines RAND's partnership with SPARK to refine the Jobs Programme's M&E system in Turkey and Jordan. Key findings include the need for tailored KPIs, enhanced local partner capacity-building and innovative data collection methods. The report provides a comprehensive framework for assessing programme effectiveness and sustainability, including recommendations for improving data consistency and accuracy.

Monitoring and Evaluation Support for SPARK's Jobs Programme

Olatz Ribera Almandoz, Fernando Esteves, Krishna B. Kumar, Jenna W. Kramer, Ben Baruch, Joanna Hofman

ResearchPublished Sep 25, 2024

Now in its fourteenth year (as of 2024), the Syrian refugee crisis stands as the world's second-largest displacement of people. With over 7.2 million Syrians internally displaced and 5.5 million seeking refuge in neighbouring countries such as Turkey, Lebanon, Jordan, Iraq and Egypt, sustainable solutions are imperative. The Netherlands-based NGO SPARK has been instrumental in addressing this crisis through its Jobs Programme, which aims to create employment opportunities for refugees and host communities. The programme focuses on four key components: Scaling Up SMEs, Matching Up job supply and demand, Starting Up new businesses, and Skilling Up employability skills, with a strong emphasis on supporting at-risk youth and women. Additionally, an integrated component, Partner's Capacity Building, enhances local partner organisations' operational capabilities and impact.

This report presents the results of a collaboration between RAND and SPARK to develop and execute a comprehensive Monitoring and Evaluation (M&E) system for the programme's efforts in Turkey and Jordan. It details RAND's rigorous approach, including benchmarking good practices, aligning with OECD DAC criteria, setting Key Performance Indicators, and creating data collection tools and guidance materials for partners. It also incorporates feedback from two workshop rounds with SPARK's M&E team and primary partners, highlighting valuable lessons learned and best practices. It includes practical recommendations for testing and refining the M&E system further, alongside materials for continued evaluation. This document aims to support SPARK in enhancing the effectiveness of its Jobs Programme, ultimately fostering economic integration and empowerment for refugees and host communities.

Key Findings

Tailored KPIs

Adapting KPIs to align with local partners' capabilities and regional contexts is crucial for effective M&E. Allowing local partners to select KPIs specific to their projects while maintaining a core set of standard indicators ensures both flexibility and consistency. This approach enhances the meaningful assessment of programme outcomes across diverse contexts.

Clear Definitions and Standardisation

Accurate and consistent definitions of key terms such as job quality, green jobs, start-ups and MSMEs are essential for reliable data interpretation and reporting. Standardising these definitions facilitates clear communication and comparability across different programme components and geographical areas, improving overall data reliability and coherence.

Enhanced Data Disaggregation

Going beyond basic demographic data to capture detailed information about businesses, job characteristics and types of support received offers a more nuanced understanding of the programme’s effects. This enhanced disaggregation provides deeper insights into the programme’s inclusivity and effectiveness for diverse target populations.

Balanced Indicator Approach

Incorporating both output and outcome indicators is necessary for a comprehensive assessment of immediate results and long-term impacts. This balanced approach supports informed decision-making and aligns programme activities with broader development goals, fostering sustainable outcomes.

Robust Data Collection and Verification

Optimising data collection practices with standardised reporting intervals and practical methods enhances data accuracy and timeliness. Coupling these practices with robust verification methods, including both formal and informal approaches, and real-time data entry systems ensures data reliability. Prioritising participant involvement and privacy fosters trust and compliance with data protection regulations, supporting effective programme M&E.

Recommendations

  • Conduct pilots tailored to individual countries and partners to rigorously evaluate the operational effectiveness of KPIs. This includes testing proposed definitions, disaggregation variables, and the efficiency of data collection and verification methods across varied contexts.
  • Invest in capacity-building initiatives to enhance the ability of local partners to monitor, evaluate and report on programme outcomes effectively. The new M&E system includes indicators to assess outputs and outcomes, aiming to empower local partners and bolster programme effectiveness and sustainability.
  • Explore innovative methods for data gathering. This could involve implementing pre- and post-intervention surveys to capture detailed insights into programme impacts over time, contributing to a more robust M&E system and supporting SPARK's sustainable and impactful results.

Topics

Document Details

Citation

RAND Style Manual
Ribera Almandoz, Olatz, Fernando Esteves, Krishna B. Kumar, Jenna W. Kramer, Ben Baruch, and Joanna Hofman, Monitoring and Evaluation Support for SPARK's Jobs Programme, RAND Corporation, RR-A3427-1, 2024. As of October 13, 2024: https://www.rand.org/pubs/research_reports/RRA3427-1.html
Chicago Manual of Style
Ribera Almandoz, Olatz, Fernando Esteves, Krishna B. Kumar, Jenna W. Kramer, Ben Baruch, and Joanna Hofman, Monitoring and Evaluation Support for SPARK's Jobs Programme. Santa Monica, CA: RAND Corporation, 2024. https://www.rand.org/pubs/research_reports/RRA3427-1.html.
BibTeX RIS

Research conducted by

The research described in this report was prepared for the SPARK and conducted by RAND Europe.

This publication is part of the RAND research report series. Research reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND research reports undergo rigorous peer review to ensure high standards for research quality and objectivity.

This document and trademark(s) contained herein are protected by law. This representation of RAND intellectual property is provided for noncommercial use only. Unauthorized posting of this publication online is prohibited; linking directly to this product page is encouraged. Permission is required from RAND to reproduce, or reuse in another form, any of its research documents for commercial purposes. For information on reprint and reuse permissions, please visit www.rand.org/pubs/permissions.

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.