Following on from my previous post, last week I had my first experience of being on a BBSRC committee, and it was certainly interesting. Main points:
1. The committee takes its responsibility seriously. IMs take time to read and understand the grants – even when on topics outside of expertise – and try to form fair and objective judgments. The process – while time-consuming (and expensive) is also fair and robust. The discussions were respectful – even when there was disagreement, the panel tried to form objective judgments based on the grants in front of them.
2. There is a scoring culture that new committee members have to adapt to. I was unsure how to calibrate from the 6 point scale the reviewers get to the 7 point scale the committee has – what was interesting is that there is a clear shared culture over what the scores – and especially in range from 5 to 6 – ‘mean’. So this means that IMs do tend to decide their own scores for grants – using the referee scores as a rough guide – rather than taking an average of referee scores. This is actually a good thing for two reasons. First, it brings consistency – if the committee members work to a mutual understanding of the scores, then they can score consistently, in a way that referees cannot, because their scores are much more individual and biased. Second, it means that what the referees actually write is more important than the score they give. A high score without clear justification is not worth much to the grant; a lower score, but with questions that are well answered by the grant authors, will not be adverse to the grant – indeed well-answered questions will help it.
So this means for reviewers: write good reviews. Whether your scores are high or low, make sure you have clear reasoning behind your score, and clear questions for the grant authors. For grant writers: don’t get too fixated on the actual reviewer score. Focus on the comments/questions and provide well-evidenced answers.
3. Most grants will have at least one IM who is not directly expert in its content. This means that it is really important for grant writers to have sections of the grant (why is the work important and what will it lead to) that are understandable by a broader range of scientists. The work plan should contain technical details – but if the whole grant is technical, it can be hard for IMs to ‘sell’ the grant to each other or the committee chairs.
4. There is a very embedded culture around preliminary data – that was clearly more manifest in the empirical biology members of the panel over the computational biology members. On some levels this is important, but on other levels perhaps it is given too much weight – to the detriment of more innovative research. Of course, it is crucial to provide evidence to support the hypotheses/assumptions of the work plan and choice of research activity. Equally, it is crucial to demonstrate that the lab can be successful in using the particular techniques, organisms/cells etc required for grant success. Without these a grant will not be successful. On the other hand, I do wonder how much value it truly is: preliminary data will have been produced by PhD students or PostDocs who most likely will no longer be in the lab by the time the PostDoc on the project starts. In the end it is the quality of that PostDoc that will make the biggest difference – their background knowledge, their ability to learn new skills, etc. A good postdoc will learn new techniques and skills; a poor postdoc will not – and none of that relates to the quality of the preliminary data. That is not factored into the entire grant process!
5. The funded grants were really awesome. It is a competitive process – and while there is some element of luck – the ones that reached the top were exceptionally good, and many that will not be funded were also excellent.