One of the ‘perks’ of my professorial promotion is that I get to partake in our school’s REF (Research Excellence Framework) readiness review – that is, I get to read a stack of papers from my colleagues, and decide whether they deserve 1, 2, 3 or 4 stars (or even no stars at all) according to a scale that takes into account novelty, significance, rigour, and ‘other’ (that sprinkling of magic fairy dust that makes a 4* paper). In doing this I realized that the process can be broken into two separate activities:
1. Reading research papers written by my colleagues, that are outside of my normal discipline. Actually, that is a really positive thing – something we should all do more often. I now know more about the research of some of my colleagues, know more about the world (some of the papers were genuinely interesting, even if outside my area of research), and have even learned some things that might be useful for my own group (an R package I wasn’t previously aware of). Verdict: let’s all make an effort to read our colleagues’ papers some more, perhaps especially those colleagues whose work we are not familiar with.
2. Grading my colleagues’ papers for REF starriness. This is plain silly, but no more silly than the real REF: I have some expertise in some of what my colleagues have written, but, in truth, I was not truly expert in any of the papers I scored – which is probably true of most of what happens on real REF panels too. Well, I completed the process, filled in the numbers, and seemed to be quite harsh in my judgment (~ a 1:2:3 ratio if 4*:3*:2* papers). How good can my scores really be? Was I really too harsh? Or is that about the right proportion? It is difficult to tell. One thing that was frustrating was realizing that I was doing this at the wrong time. When I read a paper, and, let’s say, at no point is written why the results of the paper might be important, surely it would be better to let my colleagues know before they publish it that they might want to explain why their results are important in their Conclusions section, rather than just have non-experts pass judgment after publication?
In the end, the whole exercise reminded me of a conversation I had a couple of years ago with our older daughter – who was 5 at the time – in which I explained the REF to her. “Every time we do some work, we get given some stars for it. If the work is not very good, we get one star; if it is quite good, we get two stars; if it is very good, we get three stars; if it is totally excellent, we get four stars.” Five-year-old completely understands the system, and says to me “Daddy, I will always give you four stars.” From this I learn two lessons: first, in the REF, we are treated like 5-year-olds; and second, that there are far more important things in life, so it is important not to get too emotional about the REF and its peculiarities.