New CRA Report Addresses Growing Integrity Concerns in Computing Research

By Alex Aiken and Nancy M. Amato, co-chairs of the CRA Working Group on Research Integrity

It is difficult to overstate the importance to computing research of integrity—the observation of formal rules and norms about what is acceptable in the preparation, submission, and peer review of research publications. Our computing research community relies on the integrity of authors, reviewers, conference chairs, and journal editors to function properly and to fulfill its core purpose of advancing knowledge. Recent community concerns that violations of the rules and norms of research integrity are increasing must be taken on with seriousness, urgency, and rigor.

At the direction of the CRA Board of Directors, the CRA Working Group on Research Integrity was formed earlier this year. Meeting bi-weekly from February through May, our meetings were devoted to hearing from community stakeholders, including the computing professional societies, university officials charged with investigating violations of research integrity, recent program chairs of large conferences, representatives of the major conference management systems, and the National Science Foundation.

Consisting of 17 members—chosen to broadly reflect these stakeholder groups—the charge of the working group was to:

• Enumerate the current threats to research integrity.
• Evaluate whether these threats have significantly changed in recent years and whether the community’s mitigation efforts appear to be sufficient.
• Make recommendations on best practices to mitigate those threats where there is some consensus or suggest directions for further study where no such consensus exists.

The results of the working group’s efforts are detailed in a new publication, Report of the CRA Working Group on Research Integrity, which is now available to the community on CRA’s website.

Training Materials for New Community Members

The primary overarching recommendation of the working group is for the development of training materials to educate new members of the research community on the standards and expectations for authors and reviewers, as well as the consequences for violations of those standards.

Currently, members of the research community learn what is and is not acceptable behavior from their peers and mentors. Given that the research community is extremely diverse and has grown very rapidly, there is no reason to assume that everyone is familiar with the shared standards of integrity at the start of their careers.

Subversion of Peer Review

We believe that the subversion of peer review is the most immediate and serious of the current threats to research integrity. The working group heard evidence that abuses of the peer review system have significantly increased and evolved to exploit the very different reviewing environment that exists today compared to a decade ago. Our report makes a number of specific recommendations for protecting the integrity of peer review.

Abuse of Meeting Sponsorship

Abuse of meeting sponsorship involves creating conferences or workshops that do not meet community quality standards. The working group heard evidence that the creation of substandard, but sanctioned, meetings is increasing.

The specific current issue centers on “special topic” meetings. Typically, a professional society approves an application to sponsor a one-time meeting on a research topic, only to discover later that the proceedings of the meeting, which is generally included in the society’s on-line digital library, was assembled without even minimal standards for selecting papers.

The working group recommends that professional societies develop more rigorous standards for sponsoring special topic meetings to reduce the instances of such meetings where the proceedings have very low or no standards for publication.

Additional Threats to Research Integrity

Other areas of concern for research integrity in computing research are discussed in the report without making specific recommendations. Any training materials developed for new members of the computing research community should include a discussion of the following research integrity issues:

• Fabrication and falsification of results is knowingly attempting to publish results that are not scientifically valid.
• Plagiarism is taking credit for previous work (by others or even oneself), most often by appropriating ideas or text from past publications without attribution.
• Manipulation of citation counts is inflating the number of references to a paper beyond what is justified by the scholarship.
• Ignoring the standards of authorship primarily involves including authors on papers who have not contributed significantly to the work or, conversely, excluding people who have contributed significantly from authorship.

Generative AI in the Publication Process

The working group also discussed the abuse of generative AI in the publication process, which we defined as using automated tools to substitute for human intellectual contributions in the submission or review process. Considering that this is a very recent potential threat, the working group determined that this topic needs to be revisited when the community has additional experience with the use of generative AI in the publication process.

Access and Share the Report

The potential audiences for this report include authors, reviewers, program chairs, editorial board members, developers of conference management software, professional societies, department chairs and other university administrators, research managers in industry, and funding agencies.

As co-chairs, we’d like to sincerely thank our fellow members of the CRA Working Group on Research Integrity for the considerable time, effort, and expertise they lent to this important effort. It is our hope that the community will reckon with the findings and recommendations that we detail in the report and will share it widely.

 

Alex Aiken is a co-chair of the CRA Working Group on Research Integrity, a member of the CRA Board of Directors and the Alcatel-Lucent Professor of Computer Science at Stanford University.

Nancy M. Amato is a co-chair of the CRA Working Group on Research Integrity, the Chair of the CRA Board of Directors and the Abel Bliss Professor of Engineering and the Head of the Department of Computer Science at the University of Illinois Urbana-Champaign.