Putting the value in evaluation

October 19th, 2015

Lately I’ve been reading Daniel Fujiwara’s paper about developing SROI (Social Return On Investment) which has set me thinking (really thinking) again about why as management consultants, MB Associates is such a fan of the SROI evaluation approach. And I think it is in many ways because we are management consultants.  Our work is real-life not academic.  We are not researchers, nor evaluators, nor policy advisers, though we do all these.  We combine learning about best practice with knowledge about common practice to help managers and those that advise them to improve.  So for example our new commissioning Toolkit advises ‘hope for proper timescales, but plan for quick turnarounds’. 

As Daniel pointed out in a workshop he ran with us for Happy Museum, one of the challenges for devising effective evaluation is that practitioners, managers and policy makers (or investors) have different needs, operating at a different point in the supply chain.  Simply put, practitioners need to know how to delivery effective quality at the frontline, managers, how to run an effective service, and policy makers how to choose between services to make the greatest impact on society.  (They should leave the rest  to the pracitioners).

So in response to Daniel’s paper, we agree it’s worth pursuing ways of making SROI water-tight so that it can be used to make socially responsible policy.  But we should not be talking about basic and advanced SROI, rather about SROI for investors, and SROI for managers. The problems he describes are not priorities for non-profit managers, who are grappling with challenges that are much more operational.  I’m reminded of our work for youth charity Onside, which was thoroughly serious about introducing impact measurement into all their brand new youth zones, but was overwhelmed by operational demands in what is a chronically under-funded sector.  

By way of an example, in this blog I’d like to discuss the challenge of identifying and planning costs.  Before we even think about  impact, again and again we see that non-profits don’t have either the capacity or the capability to properly cost their services.  If they don’t understand the money, how can they possibly show value for money?  The problem is endemic – the Arts Council’s evaluation guide for many years provided no guidance on value for money, but knowing how much services cost is surely vital?  Firstly for pricing them, whether that be to a customer or an investor in terms of full cost recovery.  But as importantly, for planning, evaluating, and for creating a portfolio of services that has the right balance of financial and mission based ‘return’. By coincidence, this week I heard a business analyst talk about this in relation to dairy farmers.  He pointed out that in manufacturing, you would know the cost of every biscuit off the production line. Farmers, like many voluntary services, rely on vocation to prop up a hopelessly under-funded business model. 

We are working on proper costing with our client Transported in their Creative People and Places evaluation, allocating all their investment across their hundreds of diverse events.  In another SROI on which we are mentoring, Turner Contemporary needs to allocate their delivery, staffing and building costs across a complicated array of ‘services’, from lifelong learning to a providing Margate with a sense of civic pride.  We are not talking biscuits here, but nonetheless the same principles apply.   There are three elements to knowing a service cost: 

  • 1   The direct costs of the service – for example the freelance staff and materials to deliver a workshop, followed by the overheads, which split into two:
  • 2   Variable overheads – which vary with how much of the service you deliver – for example electricity to heat the room, and  
  • 3   fixed overheads – such as the costs of opening the gallery or the salary of the director, which won’t change. 

‘Allocating’ the direct costs is obviously fairly straightforward and will probably exist as a project budget.  To allocate the other costs, we identify a ‘driver’ which we suggest should be a ‘good-enough’ way of proportioning them out.  For example, a driver might be the amount of time staff spend on the service.  Transported use online timesheet www.toggl for this.  Or the amount of building space (considering floor area and use) that the service takes over the year, an approach we are discussing for Turner Contemporary.  It needs to make sense for the service.  And whilst all costs need allocating to the service, they should be thought of differently for planning.  For example, once you have committed to running the service for a certain number of days, the ‘marginal’ cost of one extra participant only affects the direct costs – and then only the materials used.  So it is worth getting as many people as possible into that workshop.  However if bookings exceed the space and another day is needed there is a big step up in costs, and the decision would need care. 

One of the obvious repercussions of properly allocating costs is that they may then appear high.  When we did an SROI analysis for the Museum of East Anglian Life’s work-based learning programme, because the museum venue was so important to the story of change we allocated more of the museum overheads than the usual 5% of budget.  Using visitor hours as a driver, we allocated 5% of the total museum costs.   This meant an investment of £31K budget, and over half as much again of overhead at £18K.  A focus ocus on costs would help us to value these assets.  There is no doubt that improvements could make SROI a better tool for policy-making and as a result more valued when managers use it.  But lets get the basics right first. If we are to make the case properly for investing in culture, we need to recognise value in the return and the investment, by talking about the huge value of the assets as well as the outcomes or return. 

Next month, putting the value in evaluation #2, focusing on outcomes

Get in touch

You can get in touch with us via email, phone or post.

Email - info@mbassociates.org
Telephone - 01539 731889
Address - 3 Sedbergh Road, Kendal, LA9 6AD