Monitoring and Evaluation Support for SPARK’s Jobs Programme

What is the issue?

Now in its fourteenth year (as of 2024), the Syrian refugee crisis stands as the world’s second-largest displacement of people. With over 7.2 million Syrians internally displaced and 5.5 million seeking refuge in neighbouring countries such as Turkey, Lebanon, Jordan, Iraq and Egypt, sustainable solutions are imperative. In 2018, RAND conducted a detailed study of the labour markets for refugees in Turkey, Jordan and Lebanon. This study found that, despite enormous challenges, the host countries accommodated the vast influx of refugees. One of the study’s main conclusions was that refugees need not be viewed as a problem but rather as an opportunity to bolster host economies through their productive employment and entrepreneurship.

Originating in the Netherlands, SPARK is a non-government organisation (NGO) that created jobs for refugees and host communities during the Syrian crisis. SPARK recognises that access to work is crucial for the socio-economic inclusion of refugees into communities, and the private sector is vital to creating new jobs, especially through small and medium enterprises (SMEs). As such, they have pursued a four-pronged strategy, including 1) Scaling Up (expanding promising SMEs), 2) Matching Up (connecting the supply of and demand for opportunities), 3) Starting Up (creating businesses and thereby productive opportunities), and 4) Skilling Up (increasing employability skills). All four aim to create jobs, primarily for at-risk youth and women.

SPARK has experience in and commitment to Monitoring and Evaluation (M&E) of the Jobs Programme to assess progress towards its target outcomes. SPARK’s M&E aims to use the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) evaluation criteria to examine the intervention’s relevance, coherence, effectiveness, efficiency, impact and sustainability. RAND has supported SPARK’s Jobs Programme M&E efforts by helping develop an M&E system in Turkey and Jordan (which uses Scale-Up, Match-Up and Start-Up activities).

How did we help?

RAND collaborated with SPARK to assess the M&E system relative to good practices and contribute to its development. Specifically, RAND worked with SPARK to 1) finalise a comprehensive data collection plan with well-defined metrics and indicators, 2) plan data collection and storage, and 3) establish processes for verifying data veracity and maintaining quality control.

The final Key Performance Indicators (KPIs) differ for each of the Jobs Programme’s three pillars. Work in this area was guided and informed by SPARK’s Theory of Change (ToC), Systemic Change Measurement Framework, Strategy 2030 and programme objectives. The project assessed how far SPARK’s M&E plan adheres to the OECD DAC evaluation criteria (i.e. relevance, coherence, effectiveness, efficiency, impact and sustainability) and suggested modifications as necessary. While cross-country comparisons benefit from consistent metrics and indicators, we adapted them to cultural and other on-the-ground considerations. Similarly, we suggested adapting or adding metrics for different population sub-groups.

Throughout the project, we supported SPARK in developing and piloting the data collection processes necessary to gather the data from local implementation partners for populating its Management Information Systems. In addition, we contributed to the data collection instruments SPARK developed, and provided input on various aspects of data management and analysis.

To support SPARK’s co-creation and localisation ambitions outlined in its Strategy 2030, two workshop rounds were organised with SPARK's M&E team and primary implementing partner organisations to gather feedback and insights on the proposed KPI architecture. Their inputs were crucial in ensuring the relevance, coherence and practicality of the new KPI architecture, thus enhancing the overall efficacy of the M&E system.

What did we find?

• Tailored KPIs: Adapting KPIs to align with local partners' capabilities and regional contexts is crucial for effective M&E. Allowing local partners to select KPIs specific to their projects while maintaining a core set of standard indicators ensures both flexibility and consistency. This approach enhances the meaningful assessment of programme outcomes across diverse contexts.
• Clear Definitions and Standardisation: Accurate and consistent definitions of key terms such as job quality, green jobs, start-ups and SMEs are essential for reliable data interpretation and reporting. Standardising these definitions facilitates clear communication and comparability across different programme components and geographical areas, improving overall data reliability and coherence.
• Enhanced Data Disaggregation: Going beyond basic demographic data to capture detailed information about businesses, job characteristics, and types of support received offers a more nuanced understanding of the programme’s effects. This enhanced disaggregation provides deeper insights into the programme’s inclusivity and effectiveness for diverse target populations.
• Balanced Indicator Approach: Incorporating both output and outcome indicators is necessary for a comprehensive assessment of immediate results and long-term impacts. This balanced approach supports informed decision-making and aligns programme activities with broader development goals, fostering sustainable outcomes.
• Robust Data Collection and Verification: Optimising data collection practices with standardised reporting intervals and practical methods enhances data accuracy and timeliness. Coupling these practices with robust verification methods, including both formal and informal approaches, and real-time data entry systems ensures data reliability. Prioritising participant involvement and privacy fosters trust and compliance with data protection regulations, supporting effective programme M&E.

What can be done?

Based on the findings, we propose the following set of recommendations to SPARK for consideration in future developments of their M&E system:

• Conduct pilots tailored to individual countries and partners to rigorously evaluate the operational effectiveness of KPIs. This includes testing proposed definitions, disaggregation variables, and the efficiency of data collection and verification methods across varied contexts.
• Invest in capacity-building initiatives to enhance the ability of local partners to monitor, evaluate, and report on programme outcomes effectively. The new M&E system includes indicators to assess outputs and outcomes, aiming to empower local partners and bolster programme effectiveness and sustainability.
• Explore innovative methods for data gathering. This could involve implementing pre- and post-intervention surveys to capture detailed insights into programme impacts over time, contributing to a more robust M&E system and supporting SPARK’s sustainable and impactful results.

---