CECAN Webinar: Participatory Systems Mapping for Policy Evaluation

How can we simplify these complex, large maps to communicate fast and effectively with policymakers? 

This is one of the points our approach to Participatory Systems Mapping aims to address. The short answer is that we need to pull out ‘submaps’ of the full map, based on what people tell us is interesting or important, and/or based on what network analysis approaches suggest are important or interesting in terms of the structure of the maps we create. We can then use these submaps to interrogate the map more easily, generate insights and new questions, and communicate with stakeholders. The subsection titled ‘Map analysis’ in the paper we present in this talk, gives a fuller outline – https://journals.sagepub.com/doi/full/10.1177/1356389020976153.

 A really important facet of this in practice is connecting the different types of network analysis to questions that are relevant to the stakeholders. For example in a policy context: what influences my policy outcome? (upstream of an outcome) What are potential policy levers? (downstream of an influential factor which impacts many outcomes or functions) What might be potential unexpected indirect effects of my policy? (downstream of an intervention factor). More detail on this aspect of the process should be coming in a methods paper soon.

Can you say anything about how automated or manual the process of going from the map to the different ‘cuts’ through the maps? If it’s software, what software? 

Currently we run the process manually using Gephi, a free network analysis software package. However, we are in the process of automating this process using the Python package networkx. You can use many types of software to help you (we name some below), but there is clear distinction between software which helps you draw maps easily, and those which help you analyse them. No software allows you to do both easily.

How easy is it to update these maps when something changes? Do you need to do the whole process again? 

This completely depends on how you want to design your process. It is very easy technically to add, remove, or refine factors and connections in the map. But you may want to check this with other stakeholders. Of course, you will have to re-do the analysis if the submaps are affected by any changes.

In some of our work, particularly with Defra, the maps are kept as living documents by the evaluation team and are continually updated.

How to do PSM working on online meetings? // Is there a particular piece of software you use for the mapping, especially when you run a workshop online? // Thanks for a really interesting talk? Do you have advice on the best way to go about PSM virtually? Is it best for one person to ‘draw’ and other’s comment? Is there an optimal number of people to take part? Is it better to have a number of short slots or one big session? 

Some of the software we use does allow real-time editing of the maps by people on different computers. Diagrams.net and CECAN PRSM (see below) are the ones we have used. The process is quite different online however. There is a need to run this in smaller groups and then combine the maps afterwards. We have run this both by generating the factors in advance with a large group, splitting into small groups (5-6 max) to map and then merging the maps again. It can also be done sequentially. There is a need for more focused facilitation as it can be harder to tell whether people want to speak, but don’t feel confident etc. People may also feel more hesitant to draw potential links on software rather than on a physical medium. Very important to give people time to feel comfortable with the software. Perhaps on a toy map as an icebreaker. It is also noticeable that usually the whole map cannot be seen at once using this software, so there is a need to continually take people away from small parts of the map to think about longer range connections in the system. This happens more naturally with people together around a very large map.

PRSM has been developed by CECAN specifically to help with drawing system maps collaboratively during an online meeting  or workshop.  Everyone in the meeting can participate because every edit (creating nodes and links, arranging them, annotating them, and so on) is broadcast to all the other participants as the changes are made. PRSM runs in a web browser on a desktop PC or on a tablet.  It is free to use, open source and designed for those who don’t have much previous experience or skills in system map making.  To use PRSM, go to https://www.prsm.uk/

Does the paper include these example maps? Or are they somewhere else online? Really useful. 

The trilemma and RHI examples are in the paper. Other examples are in:

• Barbrook-Johnson, P., Shaw, B., & Penn, AS. (2021). Mapping complex policy landscapes: the example of ‘Mobility as a Service’. CECAN EPPN No. 18. Available at: https://www.cecan.ac.uk/resources
• Bromwich, B., Penn, AS., Barbrook-Johnson, P., & Knightbridge, J. (2020). Systems analysis for water resources: final report. Defra report. Available at http://randd.defra.gov.uk/ (search for WT1512)
• Penn, AS, Barbrook-Johnson, P (2019) Participatory Systems Mapping: A practical guide. CECAN toolkit. Available at: https://www.cecan.ac.uk/resources/toolkits/

 

Ability to genearate various sub-maps from big hairballs seems to be where your approach can really add value. To what extent are participants enabled to do this kind of filtering based analysis? 

This is actively encouraged. We usually present some initial analysis and then gather ideas from stakeholders on what else can be done, and then iterate from there. We also try to build capacity with key stakeholders to do analysis themselves. Analysis should be co-produced just as much as the underlying maps.

Thanks, that was great! I can see this angle being very useful in analysis of general policies, have you thought about how different this would look in the evaluation of actual interventions, where some of the factors will be events rather than timeless variables?

In PSM, we have not considered representing events specifically as nodes, but rather consider them by the impacts stakeholders suggest they will have on the nodes in the map. They thus become scenarios, which we can walk through the impact of, in the full or sub maps.  When using a PSM as the basis for constructing a theory of change diagram, the map is rearranged to make key causal paths from interventions towards outcomes and impacts are clearly delineated.  Events (and their immediate outputs) can now be added as nodes on the TOC, timescales considered and the diagram annotated accordingly.  A method for construction of a TOCs from a PSM is presented in Wilkinson et al (https://journals.sagepub.com/doi/full/10.1177/1356389020980493)

Positive/Negative Arrows.  Do you find it best to define what +ve/-ve means purely numerically (X increasing leads to Y increasing) or Good/Bad – so where a value going DOWN is good (such as price of a useful input). // in the map, is the link strength (strong, weak, etc) meant in a correlation-between-variable sense? Or is that compatible with a qualitative meaning or interpretation? 

This is common point of confusion we spend lots of time discussing as the start of workshops. Sometimes people think of connections in a normative way, so a positive connection thus means ‘this factor is good for that factor’, or ‘this factor influences that in a desirable direction’, and conversely, a negative connection comes to mean something like, ‘this factor pushes that factor in the wrong way’. On other occasions, there is a more subtle confusion, with positive assumed to mean ‘increasing’ and negative meaning ‘decreasing’. This can easily go unnoticed, especially when we are dealing with factors we tend to only think of as increasing (i.e. when something increases, a negative connection does mean that factor decreases another). When we come to a factor that might be decreasing, suddenly we come unstuck (i.e. that first factor decreasing, actually increases the second, with a negative connection), and people realise they had been misusing the terms the whole time.

In fact, the connections in the map are meant to represent causal relationships. These can either be positive causal connections (i.e. if A increases, or decreases, B changes in the same direction), negative causal connections (i.e. if A increases, or decreases, B changes in the opposite direction), or uncertain or complex connections (i.e. if causal relationships depends on other factors or contexts, or if the relationship is especially nonlinear).

Note, the example map with link strengths was actually a fuzzy cognitive map which you can read about here:

• Penn, A.S., Knight, C.J., Lloyd, D.J., Avitabile, D., Kok, K., Schiller, F., Woodward, A., Druckman, A. and Basson, L., 2013. Participatory development and analysis of a fuzzy cognitive map of the establishment of a bio-based economy in the Humber region. PloS one, 8(11), p.e78319

We don’t usually use link strengths in PSM in fact, although we may prune a map down to the connections which stakeholders feel are the most important.

Causal connections / connections considered to be important by stakeholders – how do you handle disagreement between stakeholders on causal connections? 

We encourage detailed discussion on the connections where there is disagreement. If it is not resolved, then we aim to preserve the information on the disagreement and capture it in the annotations on the map.

How are influential factors determined? 

Focal factors are identified first, and then things that affect them are brainstormed and consolidated. A detailed guide can be found at https://www.cecan.ac.uk/wp-content/uploads/2020/09/PSM-Workshop-method.pdf

Influential factors can be identified by network analysis. In this case we are using the term influential to mean a factor with many outgoing connections, influencing many factors in the map directly. This would be a high out-degree node in network parlance. Of course we then have to see if its influence is meaningful in context, that is, does it impact on factors which matter to stakeholders. It is that so-called “subjective network analysis” which we are developing code to do automatically. Other sorts of network “centrality” have other implications.

Hello, thank you very much for your presentation – what software do you use to carry out the systems mapping? 

There are a wide array of options, which fall into three types: general purpose diagramming software; network visualisation and analysis software; and software built for more generic systems mapping. The table below outlines some of the pros and cons of each of these and mentions some examples. In practice, you might want to use two or more pieces of software for different purposes; if you do, you should have a plan for how you will export your map from one to the other – you do not really want to have to manually create your map twice.

Option	Examples	Pros	Cons
General purpose diagramming software	digarams.net Visio Concept board Miro	Very easy to use This option is most ‘human-readable’ Maps are easily editable Maps are easily shareable  Layout easy to manipulate manually – i.e. to produce layout from workshops	No automated map analysis is possible, analysis can be conducted manually but this is time consuming and prone to human error. Exporting map data is possible but is often difficult (i.e. least ‘machine-readable’) Cost of commercial options, but there are many free and open source options.
Generic systems mapping software.	CECAN’s PRSM Kumu yED	Easy to use Simple and/or appealing browser interfaces Map easily editable Some analysis available Maps are shareable Some advanced functionalities	Stability of more bespoke and less well-used software can cause issues. Analysis options are limited, but can be added on request. Cost of commercial options, but there are many free and open source options.
Network visualisation and analysis software	Gephi R and Python packages	Full range of analysis Can automate analysis approach once developed.	Steep learning curve to use Shareable as a file, but not possible work on at same time as others Layouts generated rely on algorithms, and manual manipulation is difficult or impossible. Layout will be unfamiliar to stakeholders.

 

Really interesting presentation – thank you! Could you please talk a bit more about the data you use and how you infer and test causality? Thanks. // Do you apply secondary data to verify your systems map? 

In all of what we presented here, we used stakeholder input to build maps so do not use data or infer causality from it. We have plans to explore how we can integrate maps that are built from (most likely time series) data, using approaches such as:

• Bayesian networks (B. Aragam, Q. Zhou, Concave Penalized Estimation of Sparse Gaussian Bayesian Networks, J. Mach. Learn. Res. 16 (1) (2015) 2273–2328),
• IDA (M. Maathuis, M. Kalisch, P. Bühlmann, Estimating high-dimensional intervention effects from observational data, Ann. Stat. 37 (6A) (2009) 3133–3164),
• Iota (S. Hempel, A. Koseska, J. Kurths, Z. Nikoloski, Inner composition alignment for inferring directed networks from short time series, Phys. Rev. Lett. 107 (5) (2011), 054101.), and
• Cross-convergence mapping (G. Sugihara, R. May, H. Ye, C. Hsieh, E. Deyle, M. Fogarty, S. Munch, Detecting causality in complex ecosystems, Science 338 (6106) (2012) 496–500.).

Thank you, really useful. In terms of the big picture how early in the journey do you think we are in the UK (or internationally)  in terms of addressing complexity through evaluation? 

We think we are a long way into that journey and now at the stage where we have the ideas and tools, we just need the practice to spread. We discuss this in this introduction to the CECAN special issue of Evaluation – https://journals.sagepub.com/doi/full/10.1177/1356389020976491

Value Proposition.  Have you developed a succinct summary of WHY an organisation should invest time and effort in mapping to understand system complexity, vs the often hidden cost of NOT doing it? 

We have not produced a one size fits all value proposition, no. Though we agree this could be a useful advocacy tool. We tend to approach each (potential) project on a case by case basis, and importantly, don’t often go out asking people to use PSM, but wait for them to come to us wanting something along these lines.

I thought there was already a built in ‘R’ module for networks 

There are several, igraph is one of the most popular. The particular automation that we are developing involves combining network analysis with subjective information on factors (see the point above on influential factors). This is not currently available off the shelf.

Could you briefly explain how factors can influence themselves? 

This is actually something that is done in fuzzy cognitive mapping (the Humber map was an FCM in fact-just a useful diagram that we clearly need to update!) and this feedback loop is there to keep the value of these factors which are external drivers high during computation. We don’t tend to use this in PSM as we don’t do this sort of dynamical analysis. Dynamically however any factor that reproduces itself, like a population, could be represented this way.