Photo: Ulf Grünbaum/Imagebank.sweden.se

3. Monitoring and Impact Assessment as a transformative governance instrument

3.1. Why transformative change needs process and impact monitoring: two monitoring instruments compared

There are several reasons for the need for transformative change, two which have been addressed in the TANGO-W project:

The development of our societies towards sustainability is mainly driven by climate change
- within this field the F-W-E nexus (the scope of the TANGO-W project) is a very specific challenge, as these are basically separate fields but with potential synergies and some unexpected interactions
Sustainability needs and global economic and social development require new processes and governance models in the urban context
- These relate to different urban transformative capacity (UTC) areas
- There is a need for more effective projects (but also on routine administration which was not part of the scope of TANGO-W)

Transformation processes require a different type of management:

Goals in transformative projects often rather vague – we have a sense of the direction where we want to go, but no concrete ideas of what the future will look like.
Learning is a key outcome, not only for the participants but also for the organisations involved. Structured learning processes based on data can be very helpful in this context.
Therefore, we need an established management process that is able to (re-)define goals based on previous results and keep an eye on the long-term impact. Part of this process is what we call impact monitoring – setting impact related goals and monitor progress towards them.
It also seems to be important to establish what we call process monitoring. This focuses on specific projects and could therefore be regarded as part of project management activities. But it also focuses on the contribution of these activities to long-term impact goals, especially when they are related to Urban Transformative Capacities (UTC).

To be effective, management of transformative processes should:

Keep in mind the difference between input (resources used), output (direct project results), outcome (change within the target groups) and impact (change beyond the scope of the immediate target groups)
Focus all activities on long-term outcome or impact (and don´t forget to manage resources and immediate results efficiently)
Consider monitoring as part of the overall management process:
1. It is a structured way of reflecting on objectives, activities and results
2. It uses indicators to measure or at least evaluate outputs, outcomes or impacts
3. It accompanies several other, more unstructured ways of reflection and learning (such as lessons learned or story telling)

In the TANGO-W project we developed impact indicators at the level of urban transformation and process indicators at the level of urban living labs:

Figure 4: Monitoring a part of management cycles (Source: Manfred Kofranek, 2024)

Monitoring Goals

In relation to these different management levels, we defined different learning objectives.

Process (Project) Monitoring aims on learning in order to improve project results by setting helpful interventions:

It focuses on the immediate needs of the ULL.
It looks at UTC development in a limited scope.
It considers specific sustainability goals of the ULL.

Impact monitoring targets on learning to improve future projects and the development of new transformative capacity:

It focuses on skills for the future.
It looks at UTC development in a more general scope.
It aims at learning to set up impact monitoring in general (learn to learn).

Monitoring, as a structured way of learning, was complemented by regular Community of Practice meetings where lessons learned could be shared and discussed in a less structured way.

Figure 5: The TANGO-W reflection and learning architecture (Source: Manfred Kofranek, 2024)

3.1.1. Aligning change requires long-term objectives and indicators

Very often urban development programmes consist of more or less unrelated activities. The complexity of transformation requires activities in many different areas that are nevertheless interrelated. The results of one project might be in conflict with objectives of another project.

To avoid such ineffective situations it is advisable to have an overall transformation strategy in place that covers both, sustainability related goals and the development of UTC. UTC will enable future projects to achieve their goals in a better and more efficient way. These objectives should be common to all activities and projects or at least known to all involved.

In this way conflicts of objectives could thus be avoided and there should be a better alignment of project goals and activities.

3.1.2. Governing transformative processes requires short-term performance measurement

The complexity of transformative processes makes single projects difficult to manage. There are so many interactions and unplanned external influences. The nature of complex projects is, that they are to some extent ”unplannable”. Good project management needs to have open eyes and a good monitoring system that enables the people responsible to act quickly if something happens that threatens the project goals.

Process monitoring, which is mainly concerned with soft facts, is an important management tool. Basically it is not about measuring performance but about establishing indicators for progress elements that are considered key to overall success.

In the case of TANGO-W, because of the focus of the projects, these were UTC-related indicators. In other cases, other indicators may be more important, but in general any transformative project will have a UTC related component.

3.2. Basic elements of monitoring

3.2.1. Monitoring processes

Goals and indicators make up the monitoring system. Goals are defined as desirable states of systems, something that describes a state that we want to reach. Indicators are something that we can observe that tell us whether or not we have reached that state. Indicators can be measurements (“hard indicators”) or evaluation results (“soft indicators”). Good definitions are crucial, the more precise the goals and indicators are defined the easier monitoring will be.

In general the process of defining goals and indicators will follow this scheme (milestones):

There is a shared vision for urban development – ideally this is defined through a participatory process involving all relevant stakeholder groups.
Goals and measures are defined in order to move towards the vision. Both are integrated into a timetable – this is what we call a strategy. This strategy should be defined as soon as the vision is clear and the process should also involve all stakeholders. Field experts can be involved to ensure the feasibility of the measures.
Identify the impact of the planned actions. This step requires the involvement of field experts from the administration or public utility companies. It should be ensured that also unplanned impacts, especially in the social and governance areas, are also taken into account and that experts from theses areas can participate as well.
Finally indicators from achieving the goals need to be defined in all areas where a significant impact (planned or unplanned) has been identified. Experts must ensure the relevance, significance, validity and feasibility of all defined indicators.
Indicators can be tested by using the results of other development projects that have been completed or are still in progress.

3.2.2. How to build target matrices & Indicator schemes for impact monitoring

There is a wide variety of possible goals in the field of sustainability and UTC. It might be helpful to group them according to an impact-oriented model.

For sustainabilty-related impact monitoring we propose to use a model that has already been applied in a similar way alredy in other urban development projects (Bieglbauer et al., 2022). For UTC related impact monitoring we propose to focus on impact categories derived from the Wolfram model (Wolfram, 2016).

Sustainability related impacts are clustered according to system change areas:

Impact dimension	Definition
Economy or economic system	Order of the entirety of economic life, includes the production of goods and the provision of services as well as the efficient use of resources and labour. Orientation question: How can the basics of life be provided economically (i.e. efficiently)?
Ecology or natural living environment	The entirety of the animate and inanimate living environment of humans, which appears to be essential for biological survival. Orientation question: How can we prevent lasting damage to our biological basis of life?
Social system	The social environment of people, the totality of groupings of people that have an influence on the lives and behaviour of other people. Orientation question: How should people's coexistence change? Which values of interpersonal relationships are important to us?
Political system	The entirety of institutions, rules, procedures and actors involved in the formulation and binding solution of political problems and decision-making. Orientation question: How can political decisions be made in a participatory, effective and efficient manner?
Science and education	All institutions and stakeholders that contribute to social and individual learning, the further development and dissemination of knowledge. Orientation question: How can a learning society be supported as well as possible?
Individuals	Life contexts of individuals and groups. Orientation questions: In which direction should life change? How can the interests of individuals be supported as well as possible?

Table 1: Definition of impact dimensions according to system change areas (Source: Manfred Kofranek, 2024)

Fields of action are typically main themes of an urban development strategy. In the case of the TANGO-W project we used the fields of food, water and energy according to the project scope.

The matrix resulting from the combination of fields of action and impact dimensions is our field for the definition of goals, measures and indicators.

Figure 6: The TANGO-W impact matrix (Source: Manfred Kofranek, 2024)

UTC related impact has been clustered according to the following categories (focus areas).

Participation of citizens and different types of organizations in defining goals, planning, implementation of measures and review
Decision making within ULL during different phases of the project (who is involved, in which role, how many persons, way of deciding, etc.)
Areas covered by shared vision (different social, economic and quality of life needs)
Resources provided for different types of activities during the project (information and knowledge sharing, time for discussion and decision making, financial resources for technical and social implementation, etc.)
Results from internal reflection and review - lessons learned
Capacities developed by different participating groups and individuals (knowledge and skills, network resources/social capital, financial resources)

Monitoring has a different scope on whether you are looking at short or long term timeframes:

UTC AREA	Short term ULL Scope \| process monitoring	Long term scope \| impact monitoring
Participation	Participative structures established within project	Participative structures established within cities
Decision-making	Decision processes within project	Decision processes within cities
Visioning	Project vision (limited scope)	Overall vision (full scope)
Resources	Resources provided for the project	Resources available for future projects
Reflexivity	Learning processes within the project	Results from project reviews; learning processes established within cities
Stakeholder Capacities	Capabilities developed by project members and used within project, including a limited sustainability monitoring	Capabilities available for all relevant stakeholders in future projects, including a specific sustainability monitoring model and processes

Table 2: Comparison of process and impact monitoring scope in the UTC area (Source: Manfred Kofranek, 2024)

When defining any type of indicator some guiding questions may be helpful:

Do the indicators support the learning process of the municipality?
Will the learning lead to a change in behaviour?
If the administration reflects on these indicators, how likely is it that they will change something if the data suggest that they should?
Is it possible to collect data or evaluate improvement in a qualitative or even quantitative way? Is it possible to interpret them according to the defined goals?
Are there sufficient resources and technical means?
Is there enough competence to interpret the data well and to propose actions?
Is there sufficient motivation to monitor and learn from observations?
This is a question of attitude and organisational culture.

3.2.3. How to build target matrices & Indicator schemes for process monitoring

Some guiding questions may also be helpful in establishing UTC process monitoring:

Participation: How can the project set up participatory structures that involve citizens and different types of organisations in defining goals, planning, implementing actions and reviewing? Which level of participation will be feasible (information only, consultation, hearings during decision preparation, full co-decision)?
Decision-making: How will decision-making be organised during different stages of the project (who will be involved, how many people from which group, how will decisions be taken, etc.) ? What decision-making structures can be established during the project?
Visioning: What vision areas will be covered by the project? What changes to the vision or goals might be necessary during the project and what could be the reasons for these changes?
Resources: What critical resources (budget, human resources, knowledge) will be needed and available? What changes in social architecture and role allocation (including personal changes) might occur and what might be the reasons for change?
Reflexivity: What learning processes can be established? What important lessons can be learnt for future projects (especially in relation to UTC-related critical success factors) might be found?
Stakeholder capacities: What critical stakeholders capacities should be developed by different participating groups and individuals (knowledge and skills, network resources/social capital, financial resources) with a view to future activities?

3.3. Good practice Examples from TANGO-W

We have learnt from the 7 TANGO-W ULLs that monitoring had a variety of different scopes:

Some ULLs mainly carried out an evaluation of one-off actions, which is process-oriented, while others focused on the ongoing monitoring of indicators, which is a more long-term and impact oriented approach.
The intended long-term impact was sometimes based on a proposed continuation of the current project (with a limited thematic scope). In some cases more general indicators were defined (reflecting an open scope).
Some indicators were only project-related, some ULLs defined rather programme-related indicators (covering several and different projects).
In some cases ULL objectives were directly related to long-term impact, in which case identical indicators can be used to monitor process and impact.
In addition, we learned that input-related indicators are easy to define (which may be the reason why most projects have a narrow monitoring scope) whereas it is much more difficult to define impact-related indicators. An example is budget as an input indicator vs. increased availability of high quality food as an impact indicator.

Below we provide some examples of proposed (long-term) impacts of the ULL activities and defined sustainability impact indicators.

Impact area	Proposed impact	Impact indicators
Economy	Increased population and tax income. New jobs within sustainable focused fields. Market for local and sustainable food is established.	Number of SMEs active in a specific area. Number of related jobs. Related tax revenue.
Ecology	Improved energy efficiency in buildings. Additional energy production from renewable sources. Reduced overconsumption (water/food).	Reduced emissions, water and energy consumption, waste. Relation of consumed renewable energy to other sources.
Social	Increased social capital (social interaction, existing communities). Improved public health.	Number of people and associations active in specific communities. Reduced sick leave.
Political	Political recognition and acceptance of sustainability issued. Guidelines implemented in politics and municipal companies.	Number of climate protection agreements decided and signed with external partners. Number of municipal projects in a specific area.
Knowledge	Knowledge/expertise is available and shared (within administration and with partner organizations).	Competence level (from survey), before and after. Number of publications concerning specific topics.
Quality of life	Availability of fresh and quality food (health and well-being). Increased leisure activities for the community.	Number of apartments renovated (or families affected). Amount of organic food produced locally. Number of visitors.

Typical UTC monitoring indicators were:

Participation	Number of new participative projects or activities started with similar topics. Number (and mix) of participants in established competence group.
Decision making	Established cross-departmental/competence board. Number of jointly made decisions.
Vision	Shared vision defined (and maintained). Topic coverage in mission statement.
Resources	Defined and secured budget (for specific type of projects). Administrative staff available for those projects. Number of specialists involved.
Reflexivity	Reflection of results within the administration. Established formats or processes for reflection in innovation projects. Supporting tools established.
Stakeholder capacities	Established and promising replication of municipality and business collaboration in terms of connected/joint projects. Sharing knowledge for increasing competences.

Learnings concerning UTC monitoring

While impact monitoring in the field of sustainability is not really new, although not very well established, UTC monitoring was an open field to the participants in the TANGO-W project. In the following we summarise some of the main lessons learned from the experience of this project.

There are four areas that are more or less covered by most of the ULLs:

Changing the role of administration
- If the administration is to take on new tasks or change its role in transformation processes, there should be a clear understanding of the intended impact and what that role might look like.
- Indicators are clearly related to the implementation of this new role model, but they should also focus on the intended impact.

Monitoring of decision making can help to make things transparent and thus prevent from too much political influence.

Participative processes
- Participation is helpful in making good decisions that serve the goals of a good life for individuals and an effective and efficient administration.
- Participation is not an end it itself – indicators should focus on the impact of participative processes in order to learn which form of participation is appropriate.
- Participation is not easy to monitor if you don´t have a clear understanding of how “good” participation looks like.

Build Knowledge
- It should be defined who in the administration (which role) should have what kind of knowledge.
- It is not easy to recognise that someone or an organisation knows something. Knowledge without use of it has no impact – so focus on activities that result from knowledge. The definition of knowledge related indicators should rather address the use of knowledge (not just the acquisition of knowledge).
- Monitoring can be very helpful in supporting learning processes, especially when the transfer of (implicit or tacit) knowledge is important, which is definitely the case in transformative programmes.

Some ULLs have explicitly addressed other UTC areas, the main lessons from which are:

Vision: A shared comprehensive vision is very important for urban development (but not for most projects). Monitoring can only address certain qualities of the vision.
Resources: Budgets and manpower are easy to monitor, knowledge and cooperation cannot be assessed in a similar way, but are sometimes much more important.
Stakeholder capacities: Monitoring is only possible if you have a clear picture of the skills and resources needed (which are not available in the administration) and this depends on the administration´s willingness to cooperate with other organisations.