Piloting Post-Completion: How do you measure real sustainability?

Project sustainability- the continuation of benefits after project support has ended- receives a lot of attention in the GEF partnership, and rightly so. Creating lasting benefits is the ultimate goal of development work, and a higher bar to reach than delivering on planned project outcomes. But evidence of the actual sustainability of project benefits is also hard to come by, because monitoring systems typically end with a project. For this reason, evaluations traditionally rely on predictions of likely sustainability of outcomes as when project support ends, based on the risks to continuation of benefit.

Given the importance of actual sustainability of outcomes, and of supporting the design of projects with lasting benefits, the GEF Independent Evaluation Office (GEF IEO) developed a methodological approach for post-completion verification of the sustainability of project outcomes. The approach includes a data collection protocol and guidance to facilitate a consistent approach to gathering evidence of sustainability years after a project has ended.

This approach was piloted with on the ground for the first time as part of the 2020 Program Evaluation of the Least Developed Countries Fund. The evaluation team visited two projects: Integration of Climate Change Risks and Resilience into Forestry Management in Samoa and Reducing Vulnerability to Climate Change by Establishing Early Warning and Disaster Preparedness Systems and Support for Integrated Watershed Management in Flood Prone Areas project in Rwanda.

Photo: Samoa Ministry of Natural Resources and Environment Nursery on Savai'i, Island (project site for GEF ID 4216)

Challenges

Post completion assessment of project can pose challenges. Given the mandate to take a broader view of the portfolio of GEF's work, GEF IEO can only undertake project level assessment for a very limited number of projects. Recognizing that a few projects cannot be representative of a whole portfolio, selecting projects which can still provide illustrative evidence is a tricky task. For the program evaluation, the team first identified projects which had been completed between 3-5 years prior to the mission, with terminal evaluations available. Countries with a larger portfolio of projects were then prioritized for selection, and a conscious choice was made to pick projects in different regions. Projects with poor performance delivering on outcomes during implementation (based on terminal evaluation ratings) were eliminated as options to avoid cases where sustained benefits would be very unlikely, leaving little new to learn from field verification. Ultimately the two projects selected provided valuable evidence on the sustainability of outcomes in projects supported by the Least Developed Countries Fund, which complemented information from the evaluation's desk-based portfolio review.

Another challenge is in bringing together relevant stakeholders' years after a project has closed, and in gathering reliable data and information on project achievements, as well as related developments which may have occurred since project closure. While it may not be possible to find evidence on sustainability of all project activities and outcomes, the structured tool developed by the GEF IEO for post completion assessment provided guidance on what evidence to collect through interviews and document reviews. Close collaboration with GEF's implementing partners for arranging interviews and site visits, as well as their support in country, was also crucial.

Findings

Both projects showed evidence of sustainable outcomes, particularly those related to infrastructure, capacity building, and uptake of activities piloted. The project in Samoa made lasting improvements to the government's capacity by integrating climate information into the Samoa Forest Resource Information System, tailoring Samoa's Climate Early Warning System to provide forestry information, and training government staff. The project also undertook widescale training on agroforestry techniques with signs of continued uptake (though to a varying extent). The project in Rwanda established institutional capacity and infrastructure for forecasting and early warning systems, and farmland rehabilitation through terracing and agroforestry.

There was less evidence of sustainability of results from activities to mainstream climate change in policy and development plans. In Samoa, while the project successfully completed a new forestry policy integrating climate risks and resilience, with hopes at project closure that the policy would be passed, it ultimately wasn't. In Rwanda, project participants had largely forgotten work to mainstream climate in district development plans and the national Land Use Master Plan five years after the project was completed.

Next steps for the GEF IEO

Limitations on both international and local travel have put opportunities for further on the ground missions by IEO staff on hold, and limited movement for in country consultants as well. However, information can still be gathered through desk-based research and virtual interviews, focus groups, and surveys, and the IEO hopes to expand its coverage of post-completion project data in upcoming evaluations.