CERP REU Evaluation

About

CERP is contracted with the National Science Foundation (NSF) Directorate for Computer and Information Science and Engineers (CISE) to provide evaluation of the NSF CISE Research Experiences for Undergraduates (REU) programs, which include REU Sites and REU Supplements.

For NSF CISE REUs beginning summer of 2022 or later, CERP is providing evaluation for those programs at no cost. If you have an REU Site or REU Supplement, and you are interested in working with CERP for evaluation, please complete our interest form. After you submit your interest form, you will receive a confirmation email from someone on the CERP team (cerpreu@cra.org). If you are preparing an NSF proposal for your REU, we will also provide you with a paragraph to include in your proposal.

Submit an interest form

Meet the team

Burçin Tamer, Contract oversight

Heather Wright, Project manager

Kristi Kelly, Lead evaluator

Taniya Ross-Dunmore, Administrator

Audrey Rorrer (UNC-Charlotte), Project consultant

Evaluation Process

Depending on whether you are running an REU Site or you have an REU Supplement, we will use an evaluation method most appropriate for your awarded project and student population.

REU Sites

Four weeks prior to your REU start date, we will send PI instructions and a PI form. This form must be completed within 2 weeks.
Two weeks prior to your REU start date, we will send a suggested participant recruitment email and the pre-survey link for you to send to your REU participants. Participants will have until 1 week after the REU starts to complete this pre-survey (a total of 3 weeks). We will monitor that data collection and request reminders as appropriate.
On the last day of your REU, we will send you both the second PI form for you to complete and the post-survey link for participants to complete. These must be completed within 3 weeks after the REU ends. We will also monitor that data collection and request reminders as appropriate.
Approximately 2-3 months after data collection is complete, you will receive a report that summarizes your REU participants’ responses alongside a comparison group of REU participants at similar REU sites (depending on sample sizes).

REU Supplements

Four weeks prior to your REU start date, we will send PI instructions and a PI form. This form must be completed within 2 weeks.
Two weeks prior to your REU start date, we will send a suggested participants recruitment email and the pre-survey link for you to send to your REU participants. Participants will have until 1 week after the REU starts to complete this pre-survey (a total of 3 weeks). We will monitor that data collection and request reminders as appropriate.
On the last day of your REU, we will send you both the second PI form for you to complete and the post-survey link for students to complete. These must be completed within 3 weeks after the REU ends. We will also monitor that data collection and request reminders as appropriate.
Depending on the number of REU participants and our comparison group capabilities, we will use one of the following methods for reporting:
1. You receive a full report summarizing your participants’ responses alongside a comparison group of other REU supplements.
2. You receive a report that aggregates the participants in your department (multiple awards) together for reporting. Comparison group will include an aggregated sample of students at similar institutions.
3. You receive a report that aggregates your participant(s) with participants from similar institutions. An appropriate comparison group will be determined at reporting.

Work with Us

If you are interested in having CERP evaluate your NSF CISE REU program, please complete our interest form.

After you complete the interest form, you will receive an email from the CERP team (cerpreu@cra.org) within a few days confirming receipt of your form response. If you are preparing an NSF proposal for your REU, we will also provide you with a paragraph to include in your proposal.

Submit an interest form

IRB

Thank you for your interest in CERP’s evaluation of your NSF CISE REU project. While CERP has received IRB approval to conduct this evaluation, your institution’s IRB may require you to get separate approval to conduct the evaluation. CERP urges you to consult your institution’s IRB to determine whether they will require their own separate review.

Click here to request our IRB approved materials.

Guidelines for talking to your local IRB about the REU Evaluation

The information below is intended to support PIs with guidelines and talking points for approaching your own institution’s IRB about the REU evaluation.

The Center for Evaluating the Research Pipeline (CERP) has received IRB approval from Solutions IRB for this REU Evaluation project as an Exempt, Category 2 project. As the contracted agency acting on behalf of NSF, CERP has also received approval for the project and its materials from the Office of Management and Budget (OMB Control Number: 3145-0266; Expiration Date: 04/30/2025). All currently-funded NSF REU projects in the U.S. are eligible to use evaluation services being offered by CERP and are covered by this IRB approval.

Based on current OHRP definitions of when an institution is considered to be “engaged” in research, most institutions we work with do not require separate review and approval from their own institution’s IRB. This is because the local REU PI distributes anonymous pre-program and post-program survey links to the REU participants on behalf of CERP but has no other involvement in the research process (e.g., creation of the survey instruments, consent process, data collection, data analysis, or any other aspect of the study), and the PI will not have access to any data files or individually-identifying information resulting from this study.

If you are a PI seeking to clarify your local IRB’s requirements, here are some key points to cover:

- The Center for Evaluating the Research Pipeline (CERP) is conducting a national evaluation on behalf of the National Science Foundation (NSF) and has IRB approval from Solutions IRB as an Exempt, Category 2 project, along with approval from the Office of Management and Budget. This site is part of the national evaluation being conducted by CERP.
- The full study involves the distribution of surveys to REU participants at three timepoints: (1) prior to the start of their REU participation; (2) at the end of their REU participation; and (3) a one-year follow-up. These surveys will track changes over time among REU participants in their knowledge and skills, attitudes, and intentions to pursue further computing-related education and career pathways.

The REU PI’s only involvement in the data collection process is to distribute an anonymous survey link to their REU participants on behalf of CERP for the first two survey administrations (at the beginning and end of the REU).

The REU PI will have no involvement in the data collection for the third survey (the one-year follow-up); CERP will administer the one-year follow-up survey.
The REU PI will have NO involvement in any other aspects of the research process.
The REU PI will not receive a data file or any individually-identifying information resulting from this study. They will receive a summarized report of their REU results.
For information purposes: The national evaluation being conducted by CERP also includes a comparison group of participants who will be tracked alongside the REU participants. Importantly, this is completely separate from the data collection that is taking place with REU participants, and local REU PIs have no involvement in this aspect of the project. The comparison sample is derived from participants in previous CERP surveys who gave permission to be contacted for future data collection efforts.

If your institution’s IRB does require separate review and approval, you or another colleague will need to serve as PI for the portion of the research project that will occur with students at your institution.

Please complete the request form linked below to receive CERP-created materials that will help you with your local submission.

You can choose to receive a version of the materials that only pertains to the local implementation (recommended), or you may receive a version of materials that includes the comprehensive set of materials for the full national evaluation being conducted by CERP. Both sets will include a document “IRB resources for individual institutions,” which includes common IRB application questions and suggested answers.

If you have any other questions or need any additional information to satisfy the requirements for a particular question on your local IRB forms, please contact us at cerpreu@cra.org.

FAQ

Frequently Asked Questions about CERP REU Evaluation

What is the cost of working with CERP evaluation?

With funding through a contract with NSF, CERP’s evaluation services are free for NSF REU Sites and REU Supplements. PIs do not need to include a budget for CERP’s evaluation in their proposals.

If I work with CERP, do I need to hire another evaluator?

CERP’s evaluation fulfills NSF’s evaluation requirement, so another evaluator is not necessary.

Will CERP collaborate with other evaluators?

While CERP is happy to provide other evaluators with the methodology and copies of the survey, CERP is not directly collaborating with other REU Site and Supplement evaluators for this project.

Will CERP share data with other evaluators?

Due to privacy considerations, CERP does not share raw data collected through this project. PIs are encouraged to share the summary reports that CERP will provide each year (assuming that there are enough participant responses) with other evaluators as needed.

When I signed up to work with CERP for evaluation, I received a proposal paragraph. How do I include it in my REU Site/Supplement proposal?

The proposal paragraph that we provide should be inserted directly into the required section of your proposal on evaluation. . As it is a general paragraph, please feel free to add minor modifications to the language so that the paragraph includes any project-specific language or institutional information.

As an REU PI working with CERP for evaluation, do I need to have my own IRB approval?

CERP will have IRB approval to conduct evaluation through Solutions IRB. As an REU PI, you are encouraged to contact your IRB office to determine whether additional oversight is required by your institution. Once we have our IRB-approved materials this upcoming spring of 2022, we will supply a copy of the approval letter and templates on our website that you can use to initiate that conversation.

Can I add site-specific questions to the CERP survey?

We have some limitations in our ability to change our surveys as our project requires approval of all study instruments by the Office of Management and Budget (OMB), we can only include survey items we have submitted to them for approval. However, we will keep a record of any requested additions and will plan to review and submit changes for next year’s evaluation cycle.

Will surveys also be sent to mentors/staff?

Although official surveys for evaluation will not be sent to REU mentors and staff, CERP will collect information about the REU from PIs and other project leadership to better understand the structure of each REU. This data collection effort will be in service of providing appropriate comparison groups.

Who is responsible for administering the evaluation surveys to student participants?

Each REU project will be responsible for administering the evaluation surveys to participants. Prior to data collection, CERP will provide PIs (or the determined contact person) with all materials necessary for distribution, such as recruitment templates and the survey link. All communications from CERP will come from cerpreu@cra.org.