To continue the slow New Culture vs Old Structures blog posting series – here are some thoughts about peer evaluation and a couple of concrete proposals of how this system could perhaps be improved in context of arts and culture.

Peer evaluation is the key method for evaluating arts, culture and academic research. During the past years, there have been various efforts to change or undermine the importance of this kind of decision making process.

One motivation for this has been the search for short-term economic gain – there has been a wish to turn old institutions into streamlined flagships of ‘cultural industries’. Another motivation has been that separating disciplines into strict categories (art, architecture, design, etc) has seemed to be an ‘old-fashioned’ and far too rigid way to respond to the complexity and diversity of the creative aspirations of people today.

So far here in Finland the efforts to improve the decision-making process have focused on two options. They are both tempting due to their simplicity, but unfortunately have some serious shortcomings.

(BAD) IDEA NUMBER 1: LET’S USE NUMERIC MEASUREMENTS

Tuukka Tomperi’s blog has a great article (in Finnish) about the new scoring system for academic publishing. In future the academic journals will be rated on 4 levels, and researchers will get points accordingly:

- Level 3: 4 points
- Level 2: 3 points
- Level 1: 1 point
- Level 0: 0.1 points

It’s worthwhile to note the dramatic difference between levels 3 & 2 and level 0 (only 0.1 points).

The problem with this system is that it has been designed based on ‘hard sciences’ such as  physics or medicine, in which you can rather easily identify these different levels (with journals like Science or Nature being on the top of the hierarchy). Originally this system was not supposed to be used for humanities, since it’s very difficult to categorize the publications in a similar manner. Therefore it’s very likely that a lot of publications will fall into category 1 or 0.

When the renewal of the scoring system was prepared, if was emphasised that various academic disciplines operate on very different logics and therefore would need different evaluation systems (‘you cannot use the rules of rugby to judge soccer’). But now these differences seem to have been forgotten.

The problems of numeric evaluation were well elaborated in Seminar on Measuring the Effect of Cultural Policy, organised by Nordic Culture Point in November 2013. It’s notable that most of the presenters in the seminar were economists and/or statisticians.

In his introduction talk, Mikael Schultz illustrated this via an example:

QUESTION: WHAT IS THE MEANING OF LIFE?

ANSWER: 42

It might very well be that the true answer to meaning of life is in fact 42. The problem is that we don’t know how to interpret this answer. The same goes for all numeric values – if they are used in isolation, they do not actually properly measure any quality. As noted by Jaakko Hämeen-Anttila, since Aristotle we have known that it’s impossible to measure quality by quantitative means.

Here are a couple of slides that further illustrate the same point:
indicators

This slide is from the presentation by Clas-Uno Frykholm.

When thinking about evaluating culture, one needs to make a difference between three aspects:

- Output: Concrete outcomes such as produced artworks, or events with specific amount of participants, etc
- Outcome: The direct effect that the this cultural project has on individual people
- Impact: The impact that the project has in the society

In the end, it’s the impact is what matters. And impact is a very complex thing to measure, as illustrated by this graph:

kkn-slide-2B

This graph was shown by Trine Bille, referring to research by Dorte Skot-Hansen. I found this same image from this slideset and translated the terms to English.

At least these four different sets of issues should be taken into account if one wants to gain some understanding about the value of a specific cultural project.

The reason why simple numeric measurements are appealing is related to another (stupid) idea:

(BAD) IDEA NUMBER 2: LET’S HAVE STRONGER TOP-DOWN LEADERSHIP

Numeric evaluation of culture can be very tempting for some managers, bosses, administrators and politicians. The inspiration for this is of course the traditional view to the way private company functions. The CEO of a company has a lot of power and responsibility – he is the hero (or villain) who needs to make the tough decisions that keep the money flowing and company going.

The problem with this is of course that culture or an academic discipline is not a private enterprise. But somehow this basic fact does not seem to be understood by the current generation of politicians and other policy makers.

One of the most clear examples of this is what has happened to the role of the Finnish Cultural Institutes. Traditionally these institutes were very independent and had a mission to develop understanding and build cultural connections to a specific country or region.

These days the Finnish Cultural Institutes are a part of Team Finland. The goals of Team Finland are annually set by the Government of Finland. Quote from the site: “The aim of cooperation is to create a clear, flexible and customer-oriented operating model where projects falling under the scope of Team Finland activities are carried out in cooperation between state and private actors.”

Again – the logic of setting goals from the top (in this case from the very top, by the Government of Finland itself) can work well if one is running a private company, with a specific strategic position and product portfolio. But this logic fails completely if one tries to apply it to culture.

The only possible justification for a top-down decision making system is that the people in the top of the hierarchy have the best knowledge and skills to make the right decisions. If one applies this to culture, the assumption is that these people know what developments will happen in culture. In other words, they are ABLE TO PREDICT THE FUTURE. And this is obviously a VERY VERY SILLY IDEA.

- – -

So, after introducing the problems of these two simple approaches I would like to offer two alternative options that are based on improving the peer evaluation system rather than ignoring it. The focus here is in two questions: WHO is involved in peer evaluation process and WHEN.

PROPOSITION 1: LET’S INCREASE TRANSPARENCY AND THUS GET MORE EXPERTISE INVOLVED

If world is getting more complex, it might make sense to have more expertise involved than before, right?

I’ve been lucky enough to be involved in various peer evaluation committees for art and culture. Based on this experience I would say that being involved in these committees is the only way to get a comprehensive view of what is happening (or about to happen) in culture – especially regarding new, emerging culture that has not yet become mainstream.

The problem we have today is that only a tiny fraction of art & culture professionals have a chance to participate in peer evaluation committees. The only information that is made public are the positive results. This would be the same as if in political elections we would only get the list of winners, but not the list of all candidates. By looking only at the list of winners, one gets a very strongly biased view of the full spectrum of proposals.

If basic knowledge about all applicants of various grants would be public knowledge, various different things might happen. Individual actors might become aware that they are a part of a larger movement and might decide to form some kind of alliances. Some individuals and organisations might decide to focus on supporting the ones that are currently ignored, since this is what a lot of actors in arts and culture really want to do – to be the first ones to discover new talent. Increased knowledge about the undercurrents of culture would also be likely to influence what curators show in exhibitions, journalists write about culture and how individual people develop their careers.

It would naturally be difficult to make all information about all applications transparent, but a limited form of transparency might already have a great effect. It might be enough if a short description and a selection of keywords would be made public. The applicants could choose themselves how much or how little information they want to give out.

Making this information available would hopefully be sufficient enough to enable a constructive feedback cycle in which each evaluation round is an opportunity to fine-tune the process. It might become obvious that the peer evaluation committees lack expertise in some areas, and therefore the composition of the committees could be changed. Or if this it not possible, the applicants could more actively look for recommendations from relevant experts. And if even this does not help, then some applicants (or some alliances of applicants) might need to proactively campaign to gain a foothold in the cultural ecosystem.

On good comparison point for this might be the various online communities that rely on peer production and evaluation. Yochai Benkler’s text Coase’s Penguin is a good introduction this topic, although this article is already more than 10 years old (I would appreciate tips of some more recent good ones!).

This approach of increased diversity requires the kind of leadership exemplified by Ilkka Paananen, the CEO of Supercell who wants to become ‘the least powerful CEO in the world’.

PROPOSITION 2: LET’S HAVE A MORE FUNCTIONAL TIME CYCLE FOR EVALUATION

In the academic world, the time cycles are usually pretty long – no one imagines that one could get a PhD done in a couple of months. In contrast to this, the peer evaluation cycles in arts and culture are very dysfunctional.

About PROJECT funding:

Most of the individuals and organisation can only hope to get project funding for a time period of approximately one year. The application times are usually once per year and it takes several months to get the decisions. In practice this means that funding decisions usually arrive just a month or two before the moment when a ready piece of work would need to be presented.

It’s especially difficult to try to get funding for any larger project. One option is to take a risk and apply one big sum from just one funder, and hope that this will work out. In most cases this approach does not make any sense – the risk of a negative decision is just too great. Another option is to distribute the risk, and apply from several funding sources. This is not a very convenient strategy either, since it’s very likely that some funders will support the project and some will not. So in the end one will get only a part of the funding one would need.

In practice what people do is that they inflate the budgets and hope that they will get enough funding to realise the project at least in some extent. The funders of course know this, so they are therefore prone to give only a part of the funding that people apply for.

The outcome of the current setup is that in project funding, one gets the decisions too late, and very rarely enough to realise what one would actually want to do. So, we get a lot of mediocre projects created in haste, with mediocre resources. This is not good for anyone – not the creators of the project, nor the funders, nor the society.

About funding for ORGANISATIONS:

Some organisations are lucky enough to get stable funding for their basic activities. Once an organisation has been somehow managed to get inside a public funding ‘loop’, the amount of annual funding is likely to stay very stable, regardless what the organisation does. The only thing that usually happens is that a small slice of funding is cut from many organisations (aka ‘juustohöylä’, the ‘cheese slicer’). Only in an exceptional situation can more radical changes happen (see the previous posting in this series). Still, all organisations have to go through the effort of applying and reporting every year, and they receive very little feedback in response.

On the other hand, if you are not in the funding loop, your changes of getting in are very sparse. Then you are dependent on project funding (see above).

An improvement to this is very simple: there should be a stronger focus on medium length grants and thus medium length evaluation cycles. ‘Medium lenght’ in this case would be something longer than the current a-year-or-less time period but shorter than the ‘eternal’ one. A time span of of 3-5 years would seem reasonable – this is the duration of many scientific research grants and EU-grants. These longer grants exist in cultural funding as well, but right now these are an exception rather than rule. In some countries (at least Belgium and Slovenia) grants of this length are already used in public funding for cultural organisations.

And when it comes to policy making, if there would an evaluation cycle of 3-5 years, it would be important to make sure that in each decision making round there is some change to the status quo. In the current situation we are loosing one generation of artists after another, since most of the funding is going to the established artists and organisations who were lucky to get into the system some decades ago.

- – -

Fortunately it seems that there is hope for change – Frame Finland is organising an event today where the concept of an open call is discussed, Nordic Culture Point is thinking about ways to visualize data about their applications and Taike (Arts Promotion Centre Finland) decided to add many new subcommittees to the peer evaluation process (this involving more expertise than before).

That’s it for this time, to be continued in following postings!