CRA Best Practices Memo on Evaluating Scholarship in Hiring, Tenure, and Promotion
The CRA Board of Directors has recently released its latest Best Practices Memo, “Incentivizing Quality and Impact: Evaluating Scholarship in Hiring, Tenure, and Promotion.” Distinguishing between quality and quantity is key to promoting the future growth of the computing and information field. The memo advocates adjustments to hiring, promotion, and tenure practices as well as to the publication culture. Below is a summary of the reports main points. Click here to download the full memo.
The recommendations in the report were developed over an 18-month period by the CRA Committee on Best Practices for Hiring, Promotion, and Scholarship, led by Fred B. Schneider (Chair) and Batya Friedman (Co-Chair). The committee conducted interviews in autumn 2013 with more than 75 academic and industry computing and information unit heads to understand the issues and gain insights from practice. Preliminary recommendations were vetted with department chairs and CRA Deans at the Snowbird Conference in July 2014.
“Above all, quality and impact need to be incentivized over quantity…What ultimately should matter when it comes to hiring, tenure, and promotion is the quality of the research.”
The memo provides recommendations for both hiring and tenure and promotion cases.
Hiring Recommendation. Evaluate candidates on the basis of the contributions in their top one or two publications, in concert with the research statement and the other standard material(e.g., letters of recommendation, full CV, teaching statement) generally read by hiring committees in determining whom to invite to campus for an interview and, ultimately, whom to hire. Candidates should identify publications where they have played a significant role.
Tenure and Promotion Recommendation. Evaluate candidates for tenure and promotion on the basis of the contributions in their most important three to five publications (where systems and other artifacts may be included).¹ Tenure and promotion committees should invite external reviewers to comment on impact, depth, and scholarship of these publications or artifacts as well as the standard material (e.g., full CV, research statement, teaching statement). Some institutions might ask a candidate to suggest which publications or artifacts be considered, other institutions might leave that determination to the external reviewers. Per standard practice, tenure and promotion committees should read the external letters and the standard material in determining tenure and promotion decisions.
Implementing these recommendations will require attention to the transition for young researchers. Annual or reappointment reviews (which often occur after three years of hiring) should reflect the emphasis on quality—not quantity—and should recognize that high caliber research activities may take two or three years to come to fruition (e.g., publication or artifact deployment) and even longer for the impact to become apparent. A corollary follows: Evaluation of senior faculty similarly should emphasize quality over quantity, with incentives for pursuing greater risk-taking in research activities.
Publication Culture. Systemic changes throughout the publication culture would help to support better scholarship. For example, publishers could remove page limits for reference lists and could allow appendices for data, methods, and proofs. Editors, as appropriate, could consider longer submissions with the understanding that, in such cases, a longer review period would be likely. In addition to conferences with published proceedings, other professional gatherings (that do not publish proceedings) might be held where work-in-progress could be presented.
“The field benefits when researchers build on each other’s work…Certain publication formats and review processes, however, encourage practices inconsistent with these elements of good scholarship.”
CRA Committee on Best Practices for Hiring, Promotion, and Scholarship
Members include: Lorenzo Alvisi (University of Texas, Austin), David Culler (University of California, Berkeley), Batya Friedman [Co-chair] (University of Washington), Eric Grimson (Massachusetts Institute of Technology), Mark D. Hill (University of Wisconsin), Julia Hirschberg (Columbia University), Benjamin Kuipers (University of Michigan), Keith Marzullo (National Science Foundation and University of California, San Diego), Tamer Ozsu (University of Waterloo), Frank Pfenning (Carnegie Mellon University), Jennifer Preece (University of Maryland), Fred B. Schneider [Chair] (Cornell University), Eva Tardos (Cornell University), Jennifer Widom (Stanford University), Jeannette Wing (Microsoft Research), and Ellen Zegura (Georgia Tech).
“There is of course a long road ahead to shifting the culture and traditions of an academic field, especially because here it would mean changes to the publication infrastructure as well as reforming the value system. But our’s is a field that is used to quick response and there is every reason for optimism.”
¹For some disciplines represented in Information Schools (e.g., philosophy), the publication outcome is a book, with the expectation that one book would be in press or published at the time of evaluation for tenure.
