Excellent exploding mud pools, NZ; summer 2012
How do you measure the impact of your arts, media, story and culture driven projects?
How many people came to workshops? How many news stories ran? How many facebook likes? How many retweets? How many people saw the show or film? Unsolicitated feedback? Policy change?
How do you measure social change? Attitudinal shifts? Cultural impact? Change in a participant’s life?
Evaluation is often a requirement of funding, but it is also really useful internally when you ask the right questions and can really understand how and what worked.
You often ‘know’ what works, but it is sometimes hard to capture and articulate this gut feeling.
There are some things you can measure with figures and stats and some things that you have to capture through story, reflection, anecdotes and interviews. I am not very academically minded so sometimes I find this area tough to navigate, but I do know that action-research, reflecting on your process and changing, reviewing, updating how you work is important in honing your practice.
There are no templates for making change – if you could perfect a plan for change, you’d be pretty popular. There is always much debate as to what created the tipping points in any social movement, behaviour change or major public shift. There are so many theories about how social change is encouraged and achieved – I’m not going to try and address them all in one blog post as fascinating an area as it is. I might come back to this though!
That said there are a couple of things I firmly believe. Firstly everyone will always try and claim that their project or campaign made the change. I think you have to take a systems approach when understanding any major change. Although there is often a catalytic moment (often cultural) this usually sits against on a backdrop of diverse lobbying, good timing, zeitgeist, community opinion, viral ideas etc. Secondly culture and story are fundamental to making change as story and culture are what we do and who we are. This to me is the guts of why art and story matter; and this is what we need to get better at understanding and measuring our impacts.
One approach that I really like in analysing what worked and why is “appreciative inquiry”; solving problems by looking at what’s going right. As opposed to the concept of “debriefing” which often errs on the side of the negative ie ‘what went wrong and how can we avoid repeating that’, appreciate inquiry is an interrogation of ‘what worked and what were the conditions that allowed for success’. I highly recommend checking out some reading and methods on this approach – I use it a lot and find it to be really useful.
It seems that there is a kind of explosion of thinking and methodology in the area of impact around social change film making. I am sure that I will write about this again, but for now here are some great online resources I’ve been working my way through.
Evaluations and impact reports;
• Big hART have some great evaluations of our projects up on our website (which we really need to overhaul when we are not so busy on projects!) These are for the most part quite long and after looking at some other evaluations, particularly the Brit Doc ones, I am thinking of distilling some of these in to shorter documents. We’re also taking part in a longitudinal study with QUT, Murdoch and Durham Universities looking at the impact of our projects on young people we’ve worked with over many years.
• Brit Doc have published some great evaluations on the films An Inconvenient Truth, Gasland and Budrus among others.
• Working Films – who are an all round inspiring outfit – have made a series of films about films making social change to answer How do social issue documentary films do more than just raise awareness?
• Harmony Institute have published this review of Bully’s impact on social media.
• Pray The Devil Back to Hell film has written a report on its impact
• Participant Media – who make feature films (Lincoln, Promised Land) and docos (An Inconvenient Truth, Food Inc) and now have a TV channel – have also written about their impact
Fledgling Fund Impact Diagram
And here are some resources for measuring impact;
• Sparkwise is a new data collecting platform that is free and enables you to capture then transform your data in to graphics; I’ve just started playing with this with Queen of the Desert’s social media so I’ll keep you posted on what I think
• The Fledgling Fund has published ‘Assessing Creative Media’s Social Impact’ (PDF)
• Centre for Social Media has published ‘Social Justice Documentary Designing For Impact’ (PDF)
• Bay Area Video Coalition have just released their ‘Impact Playbook’ (PDF)
• Determining the Impact of Film and Video article regarding discussions at Media That Matters conference
• The Arts of Engagement interview with Working Films and Brit Doc about social engagement and impact
It seems that the documentary field is charging ahead in this area particularly in the USA. I am less familiar with resources for measuring the impact of community cultural development or theatre projects.
If you know of any other work in the film or CCD or theatre spaces I’d love you to share them. (Click the little speech bubbles at the bottom right of this post to leave a comment).