Evaluation Critique A Peer Support - amazonia.fiocruz.br

Evaluation Critique A Peer Support - recommend

When you submit a manuscript to a journal or to a conference, you do not know who reviews your manuscript. The intuition behind double-blind review is that it is harder to discriminate against people if you do not know their name and affiliation. Of course, editors and chairs still get to know your identity. The intuition behind open peer review is that if your reviews are published, you will be kept in check and may get punished if you are too biased. But people are concerned about their reviews or the reviews of their papers being published. There are many undesirable biases involved in a professional setting. There are other biases as well. Conventional affiliations are more highly rated than unconventional affiliations. PhD students submit their thesis for review without hiding their name. So why would we not want to hide the identity of the researchers despite the apparent advantages? Evaluation Critique A Peer Support

Evaluation Critique A Peer Support Video

Feedback and peer review

This paper analyzes the concordance between bibliometrics and peer review. It draws evidence from the data of two experiments of the Italian governmental agency for research evaluation.

The experiments were performed by the agency for validating the adoption in the Italian research Critiqie exercises of a dual system of evaluation, where some outputs were evaluated by bibliometrics and others by peer review.

The two experiments were based on stratified random samples of journal articles. Each article was scored Evaluation Critique A Peer Support bibliometrics and by peer review. The degree of concordance between the two evaluations is then computed. The results of both experiments show that for each research areas of science, technology, engineering and mathematics the degree of agreement Critqiue bibliometrics and peer review is—at most—weak at an individual article level.

Thus, the outcome of the experiments does not validate the use of the dual system of evaluation read more the Italian research assessments. More in general, the very weak concordance indicates that metrics should not replace peer review at the level of individual article. Hence, the use of the dual system in a research assessment might worsen the quality of information Evaluation Critique A Peer Support to the adoption of peer review only or bibliometrics only. This is an open access article distributed under the terms of the Creative Commons Attribution Licensewhich permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All data are available as Supplementary information.

Thay are also downloadable at Efficient implementation of a research assessment exercise is a common challenge for policy makers. Even if attention is limited to scientific quality or scientific Evalluation, there is a trade-off between the quality of information produced by a research assessment and its costs. Until now, two models have prevailed [ 1 ]: a first model based on peer review, such as the British Research Excellence Framework REFand a second model based on bibliometric indicators, such as Australian Excellence in Research ERAfor the years preceding The first model is considered more costly than the second.

In the discussion on the pros and cons of the two models, a central topic deals with the agreement between bibliometrics and peer review. Most part of the scholarly Evaluation Critique A Peer Support has analyzed the REF by adopting a post-assessement perspective [ 2 ].

Indeed, results of the REF at various levels of aggregation are compared with those obtained by using bibliometric indicators. A clear statistical evidence on the concordance of bibliometrics and peer review would represent a very strong argument in favor of the substitution of the latter with the former. However, there are two problems hindering the adoption of the bibliometric model for research assessment. The first is how to handle the scientific fields for which bibliometrics is not easily applicable, namely social sciences and humanities.

The second is how to manage research outputs not covered in bibliographic databases, Evaluation Critique A Peer Support as books or articles in national languages. In these cases, no substitution is possible and peer review appears as the unique possible tool for evaluating research outputs. As a consequence, a third model of research assessment has emerged, where bibliometrics and peer review are jointly Evaluaation some research outputs are evaluated by Critiqke and others by peer review.

Version 0.2 (accepted)

The evaluations produced by the two techniques are subsequently mixed together for computing synthetic indicators at various levels of aggregation. In reference to this model, the question of the agreement between bibliometrics and peer review has a constitutive nature. Indeed, a high agreement would ensure that final results of a research assessment—at each possible level of aggregation—are not biased by the adoption of two different instruments of evaluation. In the simplest scenario, this issue might happen when bibliometrics and peer review produce scores Evaluation Critique A Peer Support substantially agree, for instance, when the research outputs evaluated by bibliometrics receive the same score by peer review—except for random errors.

In contrast, let us consider a second scenario where scores produced by bibliometrics and peer review do not agree: for instance, bibliometrics produces scores systematically Evaluation Critique A Peer Support or higher than peer review. In this more complex case, the disagreement might not be a problem solely if the two systems of evaluation are Evaouation homogeneously, e. Even if the concordance is not accurate at the Per article level, the errors may offset at an aggregate level [ 25 ]. In sum, the agreement between bibliometrics and peer review is functional for validating results of the assessment.

Where to find me?

ANVUR tried to validate the use of the dual system of evaluation by implementing two extensive experiments on the agreement between bibliometrics and peer review, for each national research assessment of the years VQR1 and VQR2. They consisted in evaluating a random sample of learn more here by using both bibliometrics and peer review, and, subsequently, in assessing their degree of agreement at an individual publication level. In turn, this agreement would validate the use of the dual system of evaluation and the final results of the research assessements. Two of the authors of the present paper documented the flaws of EXP1 and contested the interpretation of data as indicative of a substantial agreement [ 6 — 9 ]. The present paper takes advantage of the recent availability of the raw data of the two experiments, in order to deepen the analysis and reach conclusive results on issues that had Evaluation Critique A Peer Support open due to the sole availability of aggregated data.]