Notes from the Marine CoLAB evaluation and reflection meeting at CGF in London, on the 22nd of January 2016.

Present: Louisa Hooper, Margaret Bolton, Esther Goodwin Brown, Vali Lalioti, Maja Kuzmanovic and Nik Gaffney

Marine environment is complex, multi-stakeholder, views on its values diverge. Science and Economics dominate decision making, and CGF perceives the need to make issues 'human'. The long term aim of Marine CoLAB is to improve conservation and management of oceans for human wellbeing, as well as to make connections and build relationships that would increase capacity of the sector. CGF expects that this can be accomplished through:

• Effective flows of knowledge (to deepen understanding of the role of the ocean on human wellbeing, as well as to better understand priorities for action and inspire others to act in new ways)
• Improved collaborations and shared learning across borders, sectors and disciplines (to improve sharing and managing ocean resources)
• Deployment of new skills & resources (increased effectiveness, influence, capacity, innovation, audience engagement)

Marine CoLAB was established to use a lab approach to work intensively with a small group of change-makers with skills and ambitions to increase impact of the sector.

Objectives:

• identify changemaking NGOs; participants, advisers, influencers, strategic partners, funders.
• establish a cross sector platform, which would include:
  • programme of workshops between 2014 & 2016 (to build capacity of NGOs to deliver game changing initiatives)
  • portfolio of projects; new, existing, joint, synergies with GOI (to foster collaboration and innovation and be enabled by flexible funding and evaluation schemes)
  • an evolving, learning community
• design an evaluation and dissemination plan

To meet CGF's aims FoAM proposed a programme of workshops designed to enable a learning community and foster a development of:

• a lab/platform/network for Valuing the oceans
• a portfolio of projects/experiments/activities/

The programme was developed in two phases: a scoping phase and a transitional phase towards a self-guided CoLAB.

[January - August 2015]

• Before the workshop collect a set of 'key questions' and select one core question to guide the group through the session
• Begin the workshop from personal experiences related to the core question (“worldchanging experiences”)
• Mapping what is known and what is unknown about valuing the ocean in the group
• Uncovering emerging trends important for the CoLAB [critical uncertainties: climate change and growth economy]
• Designing four alternative scenarios and answers to the core question. All focused on connections, gaps, values and innovation

• Begin with what each individual brings to the CoLAB [skills, personalities, strengths, etc.]
• Look at current challenges the individuals and their organisations are facing, to uncover patterns in the sector
• Form hypothesis based on existing challenges and form groups to begin exploring them
• Design iterative experiments to (1) test the hypotheses and (2) learn about each other and the LAB approach in practice, through short [3 months] collaborations

• Identify emerging themes [from current work and Marine CoLAB experiments]: systems change, changing perceptions, public engagement
• Continue developing existing experiments, add new ones that specifically address emerging themes
• Test experimental approach in a 'prehearsal' [an experiential futures technique], a role-playing exercise with 'antagonistic stakeholders'
• Complete design of experiments and create implementation plans

• Clarify learning from [results and process of] experiments
• Introduce collaborative processes for enabling scaling up experiments using an iterative approach, such as backcasting, outcome pathways, creative thinking [six thinking hats]
• Evaluate the experiments and Marine CoLAB scoping phase [using the adaptive action cycle technique]

[September 2015 - March 2016]

• Collaborative process design
• Connection to Marine NGO sector in Portugal
• Field-work [proposed: learning journeys]
• Important: mix of formal sessions, field trips to the sea and different forms of socialising [joint meals, long bus ride, site visits, shared experiences [exhibition, Oceanario, lighthouse…]
• Begin articulation of Marine CoLAB as a platform [melting pot, Marine Innovation Exchange, CoLAB…]
• Mapping of MarCoLAB Incubator including existing projects by participants, as well as experiments originated in Marine CoLAB, such as the SUPB Free London.
• Implementation plan for the proposal to the OAK Foundation.

• Evaluation of Marine CoLAB's first year [what worked, what needs work, what can we improve]
• Clarify mission and vision of Marine CoLAB
• Delve deeper into the values connecting people and oceans [and the values based approach]
• Map existing and emerging MarCoLAB incubator projects, with in-depth discussions of SUPB Free London and Game On

• Values
• Mission
• Lab strategy (action learning cycle)
• Incubator of experiments [including an update on SUPB Free London]

• Plan for 2016 (activities and envisaged outcomes)
• Evaluation framework for the activities in 2016
• Action learning cycle
• Experiment incubator and project consulting
• Towards a Marine CoLAB operational model

• Preparing a business and operational plan
• Handover facilitation to lab participants

• group → team
• adaptive collaboration with funders & participants
• multi-day residential trips
• connecting people
• drive to embrace big ambitions
• not just doing (experiments, initiatives), but being (a lab, a community)

• moving from big projects to agile experiments
• shortage of time, due to other commitments
• (divergent) values (based approach)

• Hypothesis: playful co-creation brings people together (established)
• Hypothesis: multi-day site-specific workshops contribute to community building (established)
• Hypothesis: systems change experiments (lab approach) lead to increased innovation (needs work)
  • maybe it would help if the 'lab approach' itself would be tested as a 'toy system': introduce model by making, testing rather than just talking about it
  • this was a longer process than expected, with more resistance than expected at the beginning.
  • it helps to have an environment in which it is safe to experiment, but this takes a lot of time to create
  • the participants had difficulty to design experiments in iterative ways
    • possibly due to experts preferring to avoid getting out of their comfort zone, especially when they don’t know each other
• Hypothesis: Ambitious projects can be gradually reduced in risk through iterative, experimental, phased approach. (the hypothesis works when applied, but it has proven difficult for the participants to design experiments rather than projects)
  • learn from failure
  • not have to go back to scratch
  • requires appropriate documentation and continuous evaluation
• Hypothesis: 2nd cycle will facilitate transition to self-sufficiency (established)
  • facilitators should become redundant
  • group takes more ownership of the process, become collectively engaged, become more self sufficient
• Hypothesis: there needs to be continuous evaluation and adaptation (established)
• Hypothesis: lab approach can get to an operational plan quicker than using other methods (to be tested and compared with other initiatives)
• Hypothesis: being able to take lab learning back into organisations (to be evaluated more formally; it appears to be happening from testimonials and informal conversations)

Methods that CGF and/or FoAM have experience with:

• In general, trying to find ways of using logic and planning tools 'in reverse' or in parallel as evaluation tools
• Direct feedback; listening circle, step-in step-back, Japanese post-its, debriefing, reflecting in a group
• Written reports: Evaluation templates, qualitative evaluation
• Interviews (sponsors, participants): impact conversations structured interviews
• Before and after: goal resolution, identifying indicators of progress an measuring them, asking before and after questions
• Appreciative Inquiry: AI focuses on positive attributions e.g. what is working well, by drawing on case examples which are examined for key themes and emerging patterns.
• Adaptive Action Cycle (what, so what, now what)
  • Ethnographic approaches: participant-observer, visual (documentaries), interviews, stories, collecting and evaluating participant reports
• Theory of change (both as design/planning and evaluation method) aka ToC. ToC can be used as learning object, providing a specific focus for feedback, evaluation and discussion around outcomes and assumptions. However, it has been critiqued as too static.
• Outcome mapping
• 'Most significant change’
• Logic analysis; logic models of projected impact, an adaptive ToC
• Participatory impact pathway analysis (PIPA); problem tree, visioning, network perspective, outcomes in a logic model
• Community based participatory models with stakeholders / groups / close others. interviews to determine impact / outcomes / experience (e.g. Community Voice)
• Contribution analysis - collective & individual contribution in networked governance
• System mapping
• FSG's work on evaluating collective impact initiatives; most significant change, appreciative inquiry, mapping
• After action reviews / 'Post-mortem' for projects; objectives,process,results,people: what worked? what didn't? what did we learn? A structured review on what happened, why and how it can be done better e.g. for use in interviews, debriefs and workshops.
• 'Counterfactuals'
• User / developer research (interviews, observation, (sensor) data i.e correlation of subjective and objective reports)
• Evaluating unknowns (with the black animals (swans, elephants & jellyfish))
• The 'ethanol method'

Challenge: finding appropriate ways of evaluating dynamic aspects of an initiative; the dynamics has to be present in the evaluation model as well.

• circular, non-linear and complex mapping (e.g. MLP (multi layer perspective - Forum for the future))
• mapping dynamics rather than static aspects subject to change.

References:

• CGF's Glossary of Terms
• Evaluating Complexity: Propositions for Improving Practice by Hallie Preskill, Srikanth “Srik” Gopal, Katelyn Mack, Joelle Cook
• Guide to Evaluating Collective Impact by Hallie Preskill, Marcie Parkhurst, Jennifer Splansky Juster