An ‘outside’ perspective on the research assessment exercise

With these interviews, we collect an ‘outside’ perspective on the research assessment exercise. Specifically, we interviewed Professor James Newell, one of the external evaluators in the Research Quality Evaluation 2004‐2010 (VQR 2004‐2010), and Professor Tony Payne, the 2008 Chair of the panel on Politics and International Studies in the UK Research Assessment Exercise on the strengths and weaknesses of this procedure.

Interview with Professor James Newell, University of Salford, UK, External Evaluator for Anvur (April 2, 2013)

IPS: Could you tell us how were you selected and what was the mandate you received?

JN: I was contacted by email by Anvur staff upon their own initiative. I think I was selected because I had already participated to a number of qualitative assessments of research projects funded by the Italian Ministry of Education/University (i.e. Prin, Firb). I was also contacted because of my research interests. Indeed, I was asked to review the scientific output of Italian scholars whose research agenda covers both Italian politics and history.

IPS: How large was the scientific output you were asked to assess and how much time were you granted for conducting the assessment?

JN: I received around 25-30 research products (i.e. articles, books, book chapters) to be reviewed within around six months (I was originally contacted in the summer 2012). After the first round of evaluation, in January 2013, I was once again contacted to review another set of research papers (around 12 pieces). I have just finished the referee process.

IPS: What are the criteria that Anvur suggested following in the research assessment?

JN: Each evaluator is asked to assess the research output based on three criteria: relevance, originality and internationalization. For each criterion, different scores are provided. For instance, relevance is ranked between 0-3 and refers to an assessment of whether the research product contributes to the advancement of knowledge in the field. Originality refers to the disclosure of new findings that can be taken up by other scholars in the community. Under the criterion of internationalization, evaluators are required to assess the positioning of the research product in the international scenario, in 
terms of importance, competitiveness, editorial spreading and appreciation from the scientific 
community. On top of these criteria, reviewers also have the discretion to add a very short commentary on the reviewed product.

IPS: I understand that your evaluation was limited, meaning that it covered scientific outputs that fall within the research area of Italian politics. However, was it possible to discern any pattern/common themes of investigation in the production of Italian scholars?

JN: I found that that one of the major areas of investigation is electoral behavior and political parties. However, as you said, this also reflects my research interests and it may not be indicative of the direction of the Italian political science scholarship at large.

IPS: And in terms of quality, what is your general assessment of Italian scholarship?

JN: I found that quality is extremely high. And I confess I was (as I always am) seriously impressed. The Italian political science community is a relatively ‘young’ one in that it was born in the 1970s. In spite of this, Italian scholars have been able to develop solid research programs.

IPS: You are also a faculty member of a UK University that is subject to the local research assessment exercise (RAE) where the evaluation process, like the one that is taking place in Italy, is Department-based and driven by ex ante specified criteria. In your view, what are the main problems in the UK RAE?

JN: From my perspective, one of the major problems in the UK is that research is assessed in terms of its utilitarian value. That is to say, one of the key criteria upon which research is ranked is its ‘impact’. This is not to suggest that practical implications are unimportant. However, research has also a value on its own.

IPS: What about the criticisms that could be raised at the Italian evaluation? Do you have any?

JN: Reviewers like me are asked to review ex post. That is to say, we review articles/books that have already been published and that have already gone through a referee process. In these cases, and especially in those cases where the output has been published in prestigious journals, the assessment cannot but be positive! Of course this is not just a problem in the Italian evaluation process but also in the UK.

Interview with Professor Tony Payne, University of Sheffield, UK,
Chair of the panel ‘Politics and International Studies’, UK RAE 2008 (30 April 2013).

IPS: The UK RAE has a longer tradition than the Italian one. When did it start and how regularly is it conducted?

TP: It began somewhat tentatively in 1987 but has since settled into an event that re-occurs approximately every five years or so. We had the first full research assessment in 1992, then in 1996, in 2001 and in 2008. The new one is currently taking place and will be completed in 2014. [The 2014 research assessment has been renamed into the Research Excellence Framework (REF)] Everyone expects to be a further REF in, say, 2020, but that has not been announced. In other words, the whole process has grown incrementally, and been adjusted as it went along.

IPS: The results of the research assessment provide a picture of the research quality of Departments as a whole and not of single researchers. Is that correct?

TP: Yes, the assessment is meant to evaluate Departments, or Units to use the RAE jargon, as whole. This means that the results that are made public are those that relate to the research profile of the Department. Of course, the way we get at these results is by accumulation of assessment of single research outputs. In general, each Department submits up four research outputs for each member of the Department. The relevant panel then reads them, assesses them and aggregates the results. However, we do not publish the results for each researcher.

IPS: Is the main purpose of the RAE/REF that of allocating public funding to the Universities with the highest scores in research quality?

TP: Yes, it is the main purpose of the RAE to identify the Universities that should be the recipient of the greatest bulk of government funding for research. The principle is that funding is allocated on excellence. Of course, establishing status, reputation and esteem is an ancillary goal of the research assessment exercise. This matters a lot because it shapes the overall standing of a Department and directly affects its capacity to recruit students both domestically and internationally.

IPS: Could you please tell us something about the procedures through which each panel is created and assess the research output in its field?

TP: The panel members are nominated by the relevant professional association. As for the panel I chaired in 2008 [i.e. Panel Politics and International Studies], members were nominated principally by the Political Studies Association (PSA) and the British International Studies Association (BISA). Associations usually nominate senior members of the profession in whom they confidence. In 2008, the members of the Panel were 16. Once the panel had been created, we wrote and published the subject-specific criteria that were to inform the evaluation drawing on general considerations set at the national level. In 2008 subject panels had considerable room of maneuver and flexibility in adapting the criteria to the specificities of the subject area. The panel members then proceeded with the assessment, which involved panel members reading and assessing all of the research outputs that had been submitted.

IPS: Were all the panel members political scientists? Did it happen that you needed the help of other social scientists (not political scientists) to evaluate the research outputs?

TP: Yes, all panel members were political scientists, assuming for this purpose that this term includes specialists in international relations (IR), some of whom, as we know, prefer to think of IR as a separate field in its own right. The RAE system was, however, very open and accommodating to the assessment of interdisciplinary work and provision existed for the ‘cross-referral’ of the assessment of the work on the edge of a subject panel’s competence to members of another appropriate panel.

IPS: What’s your take on your experience as a Chair of an evaluation panel?

TP: I’m very satisfied with the work the panel did. We took the job seriously, read and discussed all research outputs, especially when disagreement arose. In other words, it was a totally ‘in-house’ evaluation process – a process that totally preoccupied us for some 6-7 months full-time. Furthermore, as anticipated, in 2008 each panel was granted significant flexibility in translating the general criteria into the operational guidelines that informed our research assessment. This means that we interpreted the four star-ranking in a way that was suitable to the research output in the field of political science and international studies [i.e. in 2008, research outputs were graded on a 4* ranking ranging from a 4* score that indicates research quality that is world-leading in terms of originality, significance and rigueur to 1* score that indicates quality that is recognized nationally in terms of originality, significance and rigueur.]

IPS: You stressed twice the flexibility you enjoyed in 2008 in specifying the criteria for the research assessment. Has the situation changed since then?

TP: Yes, one of the major changes that are taking place with the shift from the RAE to the REF is that there will be more centralized control on the activities of the subject panels. They will all have to work to tighter central guidelines and report in emerging outcomes to so-called ‘main panels’ covering, say, pretty much the whole of the social sciences. The idea is to create greater consistency in the assessment process, but the practicalities of this may just generate a regression to the mean.

IPS: Next to this decreasing flexibility, what are, in your view, the sources of potential problems in the forthcoming REF?

TP: I think that one problem could stem from the formalization of ‘impact’ as one of the categories that informs the research assessment. The problem here is that this benchmark risks ‘unlevelling the playing field.’ Whereas it is possible to find common ground for all subjects, from medicine to physics to the social sciences, on what good research is about (including core principles such as rigor, originality and significance), it is much more difficult to compare across subjects based on the criterion of ‘impact’. Some subjects are much better placed than others in having an obvious impact. A journal article reporting on a new pharmaceutical product is likely to have a greater impact than a philosophical article. Hence, giving a formal status to ‘impact’ in the evaluation (and scoring it as the REF is currently doing) could create disparities between subjects and definitely opens up a number of challenges.

IPS: In spite of these potential shortcomings, the research assessment exercise is deeply ingrained in the culture of UK Universities. Why do you think this is the case?

TP: I think that the legitimacy of the research assessment derives from one its major strengths, namely, its peer-review nature. The process is run not by civil servants but by senior academics who have been nominated by the professional associations based on trust. Furthermore, the process is neither mechanical nor mathematical in that it is based on careful assessments carried out among peers. Of course, the fact that it is human beings who carry out the research assessment is no guarantee that no problems will ever arise. However, the fact that researchers own the evaluation process is certainly a positive aspect of the whole procedure and lends legitimacy to it.

IPS: What are the main weaknesses of the research assessment exercise then?

TP: It costs a lot of money and energy. People involved in the evaluation work full time for the assessment for quite a period of time. The other major problem is that, in the UK at least, the assessment increasingly certifies what we already know. The Departments you would have expected to perform better than others tend to demonstrate that better performance. The system rewards the winners and enables them to go on winning! By this I mean that funding for research goes to good Departments which then hire the best young scholars, allocate resources wisely among staff members, and so continue to do well in the RAE and the REF. In contrast, the Departments that before the RAE/REF had perhaps been struggling with the competition will largely continue doing so. They do not gain enough resources to make a real difference to their competitive position. This gives rise to two opposing views on the RAE/REF. On the one hand, it is argued that we should give up completely on the assessment because there is a well-established ranking among UK universities. On the other hand, however, the research assessment is a way to avoid hierarchies of reputation being frozen and thus keeping alive a need to ‘perform’ in research by the better Departments.

IPS: Do the panel members receive a fee for their service? In the affirmative, it is a ‘good’ fee?

TP: Yes, panel members were paid a fee, but only for attendance at meetings. The massive amount of ‘reading time’ involved was unpaid, which means that the RAE was in effect hugely subsidized by UK universities releasing key members of staff to its needs for considerable periods of time. No panel member would have ever taken on the huge amount of extra work involved merely for the fee earned!

IPS: What is your final consideration on the research assessment exercise that could be relevant for a country like Italy that has just started experimenting with its own research assessment?

TP: My overall assessment on research assessment exercises is positive. As usually the case, however, what is good when it is a moderate thing may risk becoming a negative thing when you overdo it. For instance, this is the situation we are confronting in the UK at the moment with a REF that has become increasingly complex and open to game-playing. Under these conditions, the risk is that research will not be driven fundamentally by the joy of producing a new theory or new results but just by a managerial concern on the part of Universities to do well in competitive research assessment.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Basic HTML is allowed. Your email address will not be published.

Subscribe to this comment feed via RSS

%d bloggers like this: