Evaluation of development paths in complex systems
Episodio 10: Retos de la política educativa
16 diciembre, 2020
Temporada 2, Episodio 1: Vacunación y Evidencia
20 enero, 2021
Figure 1: Interactions between the 32 system conditions
This text briefly explains how to evaluate the contributions of a program or project to long-term systemic transformation processes. The approach taken is not to evaluate a complex project or program, but rather to evaluate how the project interacted with a complex system. The approach adopted here seeks to model the phenomenon intervened by the program to identify which factors affect the trajectory of the phenomenon. We do this by recognizing that the interactions between the components of the phenomenon increase its complexity. We then investigated to what extent and in what ways the program interacted with the phenomenon and to what extent the program contributed to redirecting the trajectory of the intervened system towards the desired transformation. We will explain the use of the approach with the evaluation of the UNIDO/SECO SMART-Fish project in Indonesia (UNIDO, 2019). We present here a summary of the article Development Trajectories and Complex Systems-Informed Theories of Change, which was published in September 2020 in the American Journal of Evaluation (Zazueta, et al, 2020). A free unedited version of the article is available here.
The long-term objective of the SMART-Fish program was to support the transformation of fisheries to make them more efficient and environmentally sustainable in order to increase the generation of value in the different links of the value chain, especially among small-scale fishermen and fisherwomen. The project addressed three value chains that have different ecological, economic, and social characteristics: pangasius, tuna line and seaweed. The evaluation initially developed a model of the conditions conducive to the desired transformations.
In a first phase, the project management team and the evaluation team developed the model of the fisheries value chain system in Indonesia. The first phase consisted of five steps:
1. From a review of the technical literature, the teams identified five major domains relevant to fisheries value chains: political-regulatory, institutional, technological, financial and socio-cultural.
2. Then, a brainstorming session was held to identify the four or five most important enabling conditions in each of the five domains. 32 conditions were found.
3. Subsequently, there was an analytical review of the outcome of the brainstorming and the regrouping of the 32 conditions into six domains: 1) governance, 2) production, 3) trade and markets, 4) finance, 5) science and technology, and 6) quality and standards (Figure 1).
Figure 2: Contributions to the five most influential conditions in the system
Figure 3: Contributions for change in the domains
In phase two, the evaluation team convened three stakeholder focus groups for each value chain. To evaluate changes in the project, each group was asked to rate the status of the 32 conditions before the project began and at the present time, it was the end point of the project. The groups were then asked to rate the extent to which SMART-Fish contributed to changes in these conditions.
By adding the responses to the three value chains it was found that changes in conditions conducive to long-term objectives were most pronounced in the domains of trade and markets, governance, and production (Figure 3). These were three domains on which the project focused and where participants indicated that SMART-Fish made substantial contributions. While stakeholders acknowledged the project's contributions in science and technology, progress in many of the favorable conditions in this area was considered low. The participants also considered that there was significant progress in the conditions related to the financial area, but without any link to the project.
The evaluation team also asked the focus groups to what extent the project had contributed to the five key enabling conditions with the most influence on the system. Important advances were made under the following conditions: Policies conducive to a desired trajectory; Robust science and technology; Awareness and a shared understanding of challenges and opportunities. Of these achievements, the program made a significant contribution in the first two only. Also, although the program made important contributions to the enabling condition related to the capacity for innovation in science and technology, little progress was made in other conditions pertaining to the domain of technology.
Finally, it was concluded that fisheries-related value chains in Indonesia exhibit a high degree of complexity. The approach adopted in this evaluation allowed us to develop a model of the system intervened by SMART-Fish that allowed us to identify the conditions with a high probability of influence on the set of interactions operating within the system. Once we identified the conditions that have the greatest influence on the system's trajectory, the evaluation team inquired into the extent to which these conditions had changed.
Subsequently, it was possible to inquire into the degree to which such changes were influenced by the program. This allowed the program’s contributions to be identified on a development path consistent with the long-term objectives of the intervention and, in this case, of the Indonesian government. The approach adopted also allowed the integration of technical knowledge with local knowledge through participatory methods that promoted dialogue between technicians and scientists with other stakeholders.
UNIDO. (2019). Independent Terminal Evaluation Indonesia SMART-Fish Increasing Trade capacities of Selected Value Chains within the Fisheries Sector in Indonesia. United Nations Industrial Development Organization. https://www.unido.org/sites/default/files/files/2020-01/120110_Indonesia_SMART%20Fish_Terminal%20evaluation.pdf
Zazueta, A. E., Le, T. T., & Bahramalian, N. (2020). Development Trajectories and Complex Systems–Informed Theories of Change. American Journal of Evaluation, I–20. https://doi.org/10.1177/1098214020947782
