jump to contents

CIVR VQR

CIVR VQR 2004-2008

» HOME
Italian version

The CIVR

Contacts:
-email: civr@cineca.it
-phone:+39 (0)516171839

The VQR

2004-2008 VQR Guidelines

Emblema Repubblica Italiana
Ministero dell'Istruzione, dell'UniversitÓ e della Ricerca

2004-2008 VQR Guidelines

FORASMUCH AS the law on July 14th, 2008, no. 121 "Conversion into Law of the decree on May 16th, 2008, no. 85 including urgent rules for adaptation of the State Structures in application to art.1, paragraphs 376) and 377), of the Law on December 24th, 2007, no. 244" and, in particular, art. 1, paragraph 5);

FORASMUCH AS the Legislative Decree no. 204 on June 5th, 1998 and following modifications and integrations with which dispositions for coordinating, planning and evaluating the national policy related to scientific and technological research were released;  

FORASMUCH AS the Ministerial Decree no. 643 on November 24th, 1999, concerning the organization and functioning of the Committee for Evaluation of Research (CIVR);

FORASMUCH AS the DPCM on September 7th, 2007 with which the Committee for Evaluation of Research has been re-established;

FORASMUCH AS the DPR on January 20th, 2009, no. 17, "Regulation including dispositions of MIUR re-organization", and the DM on July 27th, 2009 of "Identification of the managing offices of non-general level of the central administration of the MIUR";

FORASMUCH AS the Ministerial Decree on December 29th, 2008, no. 427, registered at the Court of Counts on June 19th, 2009 (reg. 5, paper 137), with which the 2004-2008 Evaluation program was approved;

FORASMUCH AS the economic-financial executive plan of the Five-Year Research Evaluation Exercise (VQR) concerning the period 2004-2008;

FORASMUCH AS art. 2, paragraph 1), of the Decree on November 10th, 2008, no. 180, converted into Law on January 9th, 2009, no. 1, that rules that, starting from 2009, in order to promote and increase the quality of the activities of state Universities, and in order to improve the effectiveness and efficacy of resource utilization, an amount not less than 7 percent of the ordinary fund according to art. 5 of the Law on December 24th, 1993, no. 537, and following modifications of the extraordinary fund according to art. 2, paragraph 428), of the Law on December 24th, 2007, no. 244, with progressive increases in the following years, is released to state Universities based on the quality of education and the results of the educational processes, and the quality of scientific research;

FORASMUCH AS the necessity to set rules and procedures for starting the process of evaluation according to the general principles outlined by the Guidelines for Research Evaluation by the CIVR and to identify the necessary instruments and resources for its creation;

FORASMUCH AS the opinion of the Conference of Presidents of Italian University (CRUI);

FORASMUCH AS the opinion of the National University Council (CUN);

D E C R E E S

Art. 1 - Structures
1. The present Decree regulates, through the criteria and modalities hereafter reported, the Evaluation process involving the research results obtained from the following structures (hereafter called ‘Structures’):
a) state Universities;
b) non-state Universities that grant degrees;
c) public research Institutions controlled by the MIUR (Ministry of Education, University and Research), except for Institutions working only as agencies;
d) other public and private Structures involved with research activities, which ask to be evaluated and which pay for the relative expenses.

Art. 2 – The Evaluation process
1. The Evaluation Exercise starts after a specific Call for Submissions by the President of the Committee for Evaluation of Research (CIVR).

Art. 3 – General organization
1.Evaluation of Structures is organized in evaluation areas, and in the present Decree they will be called ‘Areas’, which correspond to the 14 scientific-disciplinary areas of the National University Committee (CUN)

2. For each Area a Panel is set, made by highly qualified Experts, also foreigners, chosen according to their scientific expertise and previous experience with evaluation procedures.

3. Panelists cannot be more that 540; they are proposed by the CIVR and nominated by the Minister.

4. For those Areas characterized by a particular disciplinary heterogeneity and high number of publications to be evaluated, the CIVR within the Panels can set sub-Panels with specific disciplinary expertise; however, the total number of Panelists must remain fixed, as reported in paragraph 3).

Art. 4 – Duties of the Structures: human resources
1. The Evaluation Exercise involves the research personnel of the Structures (both permanent and non-permanent) who produce publications (which represent measurable items). Such personnel includes both professors and researchers working in Italian Universities (including assistants), researchers working in research Institutions and other Structures (all of them will be called ‘researchers’).

2. For the aims of the present Evaluation, Structures utilize the information in the data banks of the MIUR. The Structures themselves are obliged to certify the information provided by the Ministry within 45 days from the publication of the Call for Submissions according to art. 2. In particular, they must certify the list of the names of the researchers and of the technical-administration staff (including social and health personnel) both permanent and non-permanent, working for the Structure on the date of publication of the Call for Submissions according to art. 2, also reporting the Areas and the Departments in which such personnel belongs. For the non-state Universities an ad hoc procedure is available for delivering the aforesaid data.

3. Upon CIVR judgment, other functional organizations inside Structures can be considered as being similar to Departments.

4. For the aims of the present Exercise, researchers belonging in the various Structures are expressed as Full Time Equivalent (FTE), In particular:
a) each University permanent and non-permanent professor and researcher (except for only teaching personnel) weighs 0.5 FTE, without distinction between full time and part time. Also research Institution researchers weigh 0.5 FTE whenever besides research they carry out also other activities (e.g. health care, etc.);
b) each permanent and non-permanent researcher belonging in research Institutions or in any other Institutions weighs either 1 FTE if full time or 0.5 FTE if part time;
c) each technologist belonging in either research Institutions or in any other Institutions weighs 0.5 FTE;
d) each University permanent professor and researcher who has been affiliated to either a research Institution or to any other Institutions for at least 2 years weighs a further 0.5 FTE for the activities carried out in these Institutions (total 1FTE).

5. Like for researchers, Structures verify within 3 months from the publication of the Call for Submissions according to art. 2, the list of the names of the training personnel, such as:
a) PhD personnel in Universities for the academic years 2003-04, 2004-05, 2005-06, 2006-07, 2007-08, specifying if granted with scholarships or not, and reporting the Area, the Department and the funding bodies;
b) post-doc personnel with scholarships granted by the Structure, by December 31st, of the years 2004, 2005, 2006, 2007 and 2008, reporting the Area, the Department;
c) personnel granted with research funds by the Structure by December 31st, of the years 2004, 2005, 2006, 2007 and 2008, reporting the Area, the Department;
d) medical school post-graduates for the years 2004, 2005, 2006, 2007 and 2008. For these an ad hoc procedure is available for data delivery.

Art. 5 – Duties of the Structures: selection of publications
1. The following publications are considered for Panel evaluation:
a) journal articles (only those with ISSN);
b) books and their chapters, including conference proceedings (only those with ISBN), as well as critical editions and scientific comments;
c) deposited patents;
d) compositions, drawings and design; performances, shows and expositions; manufactures and art operas, and their projects; also data banks and software, on-line cards and maps (only if with publication that enables an adequate evaluation).

2. The following publications are not considered for Panel evaluation:
a) purely editorial activities;
b) letters, corrections and conference abstracts (even if published in journals);
c) texts or software for teaching purposes;
d) trials and routine analyses;
e) internal technical reports.

3. Each University researcher, according to art. 4, paragraphs 4), letters a) e b) –only if non-permanent- and paragraph 4), letter c) belonging in a Department (or in any other similar organization) on the date of publication of the Call for Submissions according to art. 2, inserts in the data bank set by the CINECA according to art. 13, paragraph 1), at least 2 publications of which he/she is author or co-author, for the five years, and ordering them by scientific importance. Each research Institution full time researcher according to art. 4, paragraph 4), letter b) submits at least 4 publications. Each researcher working for both University and research Institution or any other Structure, according to art. 4, paragraph 4), letter d), considering the additional activity carried out, submits at least two more publications for the five years, with the affiliation of the research Institution or any other Structure. Researchers not submitting publications are considered as being inactive; those submitting a number of publications which is lower than the one required are considered as being partially inactive.

4. By the date of publication of the Call for Submissions according to art. 2, the Structure send to the reference Panel on line a PDF of the publications that are going to be subjected to evaluation, selecting them based on the following modalities:
a)  Universities select two of the publications submitted by each permanent and non-permanent researcher;
b) research Institutions and other Structures select 4 of the publications submitted by each full time permanent and non-permanent researcher, and 2 for each part time permanent and non-permanent researcher; they also select 2 publications submitted by each researcher according to art. 4, paragraph 4), letter d).

5. Each selected publication must include only an illustration chart, reporting:
a) reference data;
b) name, last name, gender, position and Department of all authors affiliated to the Structure on the date of publication of the Call for Submissions according to art. 2;
c) Area and Sector (SSD);
d) an abstract of the publication;
e) whatever information could be useful for defining the international ranking, notoriety of the publication, as well as the authority of the publication means, Italian and international recognitions, and for patents, also notes on users and on socio-economic fall-outs should be reported;
f) key words (no more than 5).

6. The Structures guarantee access to the selected publications by Evaluation bodies.

Art. 6 – Duties of the Structures: other data and information
1. The Structures deliver, on-line, within 6 months from the publication of the Call for Submissions according to art. 2, other data and information separated by Area and Department, concerning:
a) patents and spin-offs entitled to or co-entitled to the Structure, specifying in case of spin-offs, duration and trend;
b) cash flow derived from patent selling and licensing, with information concerning buyers;
c) permanent and non-permanent researchers affiliated to the Structure but visiting foreign Institutions, and researchers belonging in foreign research Institutions visiting the evaluated Structures for more than 3 consecutive months, during the five years;
d) cash flow derived from research projects funds, for each year for the five years, specifying the projects funded by PRIN, FIRB, FAR, Square Programs by the European Union and the European Research Council and other public and private subjects (Italian and foreign), including the funds obtained by the non-permanent researchers belonging in the Structure;
e) funds and co-funds (for the five years), given by the Structure to research projects, separated by Area, specifying the amount obtained through contracts and agreements.

Art. 7 – Internal Evaluation Committee/Nucleus
1. The Internal Evaluation Committee/Nucleus, or anyone who is responsible for the Structure, writes a report on the Structure concerning the years 2004-2008, to be sent via line within 6 months from the publication of the Call for Submissions according to art. 2, including the following issues:
a) methodology used by the Structure to select the publications, and differences with the previous Evaluation Exercise;
b)  critical analysis of the links between internal research evaluation and decisions taken for resource allocation, with specific references to the Guidelines of the 2001- 2003 VTR.

Art. 8 –Panel duties
1.  Panels have the responsibility to evaluate the quality of the scientific publications submitted by the Structures, to set a ranking list of the Structures in each Area. In order to judge the quality of the publications Panels adopt either one or both of the following two methodologies:
a) citation analysis (where possible) carried out directly by each Panel, which uses for that aim the data banks selected with the CIVR;
b)  peer-review, carried out by Experts chosen by all Panelists (no more than two); Experts will anonymously express the judgment on the quality of the publications subjected to evaluation.

2.Each Panel, at least one month before the end of the selection procedures, according to art. 5, paragraph 4), communicates the methodologies to adopt, taking into consideration both the procedures used and accepted internationally and the disciplinary specificity.

3. Analysis of the following aspects is part of the judgment of quality:
a) importance, as added value for advancement of knowledge in the field and for science in general, as well as the derived social benefits also in terms of consistency, effectiveness, promptness and duration of the fall-outs;
b) originality/innovation, as contribution to advancement of knowledge or to new acquisitions in the field;
c) internationalization and/or international stand as positioning in the international scenario, in terms of importance, competitiveness, editorial spreading and appreciation from the scientific community including explicit cooperation with researchers and research groups from other countries;
d) concerning patents, the judgments must also include notes on technology transfer and development, and socio-economic fall outs (even potential).
For the publications done in cooperation, Panels can weigh the contribution based on the role played by the participants, valuing if first or last author, if corresponding author, based on the standards of the different scientific communities.

4. The descriptive judgments must translate into synthetic judgments, and all publications are placed in one of the following levels of merit:
a) Excellent: the publication is in the higher 20% of the value scale shared by the international scientific community (weight 1);
b) Good: the publication is in the 60% - 80% segment (weight 0.8);
c) Acceptable: the publication is in the 50% - 60% segment (weight 0.5);
d) Limited: the publication is in the lower 50% (weight 0);
e) Not assessable: the publication is not in the typologies included in the present Exercise; it represents a case of fraud or plagiarism; it includes attachments and/or files that are not adequate for evaluation; it has been published during the previous or following years with respect to the present Exercise (weight -1).
Based on art. 5, paragraph 3), for each missing publication with respect to the number of expected publications a negative weight will be assigned (weight -0.5).

Art. 9 – Panel final report
1. Panels write the final report, and deliver it to the CIVR within 12 months from the publication of the Call for Submissions, based on art. 2. The final report includes the following 3 parts:
a) Final evaluation of publications (Consensus report). For each publication the Panel writes a final judgment of merit, based on 3 descriptive levels reported in art. 8, using the evaluations by the Experts and/or the citation analyses. Publications that were carried out in cooperation with different Structures and submitted in different Areas are subject to only one evaluation carried out by a specific inter-Area Consensus groups. The evaluation obtained by each single publication is not be spread publicly.
b) Area ranking list. Each Structure obtains a score derived from the sum of the weights given by the Panel to each publication, taking into consideration also the weights of the publications that are missing and not assessable. Structure ranking is by size segments. For each Structure a score is also identified and processed as stated before, which is specifically referred to the subset of new researchers introduced in the Structure, and also to the promotions obtained by researchers in the Structure during 2004-2008. Such identification is not done if the subset of researchers includes less than 3.
c) Area final report. The President of the Panel and the other Panelists write the Final Report as follows:
i. methodology adopted and work organization, including resolution of any possible conflicts of interest of Panelists;
ii. Area evaluation and analysis of the strong and weak points, in relation to quality, amount, and ownership of the publications;
iii. specific section for patents, highlighting the socio-economic fall-outs, even potential.

Art. 10 – Bibliometric analysis by the CIVR
1. In order to assess ranking of Italy as compared to other countries, the CIVR in agreement with the Panels carries out, where possible, citation analyses of all the publications produced in the five years, and evaluated according to the specific Areas and categories. 2. Citation analysis is carried out considering the date of publication of the Call for Submissions according to art. 2, based on the criteria defined with the Panels. 3. The results of the analysis based on the previous paragraphs are published simultaneously as those reported by the Panels.

Art. 11- Final Report by the CIVR
1. The CIVR writes the Final Report of the VQR and delivers it to the Minister within 18 months from the publication of the Call for Submissions according to art. 2, as follows.

2. Evaluation of the Italian Research System (SNR)
a) Evaluation of the SNR as a whole is done by Areas and is based on the integration of the analysis of all the elements available.
b) Concerning the specific Areas and Categories, the CIVR can utilize citation analyses to compare Italy with other countries.

3. Evaluation of Structures
a) Structure evaluation is based on the Panel Reports, as well as on all the data and information delivered to the CIVR by the Structures themselves. In processing the Judgment of merit, the CIVR adopts the following criteria, with their relative weights:
i. Quality of the evaluated publications (weight 5). Performance is measured by the total rating obtained by all the publications submitted by the Structure to Area Panel evaluation, taking into consideration also the publications that are missing and not assessable.
ii. Ownership of excellent publications (weight 1). Performance is calculated multiplying the number of excellent publications by their mean degree of ownership, which is expressed by the relation between the number of authors belonging in the Structure/Department (also considering multiple affiliations) and the total number of authors of the publication. Authors are considered as being affiliated to a Structure/Department when they belong in the Structure/Department (independently of the contract conditions) on the date of publication of the Call for Submissions according to art. 2.
iii. Inclination to research training (weight 1). Performance is measured on the mean annual number of students (PhDs, scholarships, post-docs and post-graduates from medical schools).
iv.  International mobility (weight 1). Performance is measured on the mean annual number of person years of the permanent and non-permanent researchers in and out of the Structure for more than 3 consecutive months.
v. Capacity to use Structure own funds for research (weight 1). Performance is measured on the mean annual amount of funds derived from untied funds of the Structure. The numeric value of the aforesaid indicators is expressed as percentage of the total value of the Area and as percentage of the deviation from the expected value determined using the number of FTE researchers in the Structure, Area by Area on the date of publication of the Call for Submissions based on art. 2.
vi.  Capacity to attract research funds (weight 1). Performance is measured on the mean annual amount of funds derived from the following research projects:
1. from PRIN, FIRB and FAR;
2. from Square Programs of teh European Community and the European Research Council;
3. from other public and private subjects.
The numeric value of the indicators is expressed as mean of the percentages of the total values of the Area of each type of funding and as percentage of the deviation from the expected value determined using the number of FTE researchers in the Structure, Area by Area on the date of publication of the Call for Submissions based on art. 2.

b)  Area performance of the single Structure is expressed as percentage of the total value of the Area and is derived from the total score obtained by the Area in application of criteria i)-vi) reported in the previous paragraph.

4. Evaluation of Departments (or of other similar organizations).
a) For the judgment of merit, the CIVR adopts the following criteria:
i. Quality of the selected publications. It is indicated by the total rating obtained by all the publications selected by the Department and submitted to Area Panel evaluation, taking into consideration also the publications that are missing and not assessable.
ii. Ownership of excellent publications. It is given by the relation between mean degree of ownership of the publications of the Department judged as excellent and that of the whole Structure.
iii. Inclination to research training. It is given by the relation between the mean annual number of students (PhDs, scholarships and post-docs) in the Department and that of the whole Structure.
iv. International mobility. It is given by the relation between in-mobility and out-mobility of the Department and that of the Structure.
v. Capacity to have access and to attract research funds. It is given by the relation between the mean annual amount of funds from research projects (e.g. PRIN, FIRB and FAR, Square Programs from the European Union and from the European Research Council, from other public and private subjects) obtained by the Department and the corresponding total amount of the Structure.
vi. Active researchers. The relation between the weighted sum of the active researchers (weight 1) with those partially active (weight 0.5) and the total number of researchers in the Structure on the date of publication of the Call for Submissions based on art. 2 is assessed.
vii. Technology transfer activities. Performance is measured by the total score obtained from technology transfer activities and research upgrading of the Department.
The numerical value of each of the aforesaid indicators is expressed as percentage of the total value of the Structure and as percentage of the deviation from the expected value determined on the number of FTE researchers in the Department on the date of publication of the Call for Submissions according to art. 2

5. Evaluation of technology transfer activities (patents and spin-offs).
a) Each Structure is evaluated in relation to the following parameters:
i. application for Italian patents during the five years (weight 1);
ii. patents as PCT (Patent Cooperation Treaty) during the five years (weight 1);
iii. patents granted in Italy during the five years (weight 1);
iv. patents granted in Europe, in USA and in Japan during the five years (weight 2);
v. patents that are active on the date of publication of the Call for Submissions based on art. 2 (weight 1 for those granted in Italy; weight 2 for those granted in Europe, USA and Japan);
vi. cash flow from patent selling and licensing during the five years (weight 2);
vii. number of spin-offs activated during the five years (weight 3).
The numerical value is expressed a percentage of the total Area value and as percentage of the deviation from the expected value determined on the number of FTE researchers in the Structure, Area by Area, on the date of publication of the Call for Submissions according to art. 2.

Art. 12- Transparency
1. Based on art. 6, paragraph 4), of the Legislative Decree 204/98 and following modifications, the MIUR can communicate and spread all the results of the 2004-2008 VQR, including the judgments of the single publications evaluated, it being understood the anonymity of the Experts.

Art. 13 – Informatics system and economic-financial aspects
1. Based on the Ministerial Decree on December 29th, 2008, reported in the Introduction Section of the present Decree, the informatics system as well as the accounting procedures of the present Evaluation Exercise are carried out by the Inter-university Consortium CINECA, based on a specific agreement.

2. The Components of the CIVR will be paid according to the Ministerial Decree no. 643 on November 24th, 1999.

3. Panelists will be paid like the members of the Guarantee Commissions for the selection of university research projects particularly important for Italian interests. The remaining subjects involved with the present Research Evaluation Procedures will be paid according to the Inter-ministerial Decree no. 69 on April 9th, 2003. Art. 4 of the aforesaid Inter-ministerial Decree applies to all individuals involved.

4. The total expense, based on the costs included in the economic-financial plan cited in the Introduction Section of the present Decree involves, for state Universities, the annual funding by the MIUR in favour of the CINECA, which carries out also the accounting procedures of the present Evaluation Exercise, and the funds provided by the other Structures involved in the Evaluation Exercise.

5. The Evaluation activities based on art. 1, paragraph 1), letters b), c) e d) will be paid by the respective budgets.

Art. 14 – Technical-scientific coordination and support
1. Based on art. 14, paragraph 2), concerning the ANVUR, the activities based on art. 1 of the present Decree, are by the Committee of the Agency, except for the technical-scientific coordination of the Evaluation activities that are directed and headed by the President of the CIVR.

2. The support to the CIVR activities for the aims of the Evaluation Exercise is provided by the III Office of the Department for University, Higher Art, Music and Dance Education, and Research of the Ministry of Education, University and Research (MIUR).

Roma, March 19th, 2010

THE MINISTER

CIVR - Committee for Evaluation of Research
VQR - Five-year research evalutation (VQR) - 2004-2008 exercise