Expanding the Pipeline: Celebrating 10 Years of the CRA Data Buddies Project

By Heather Wright, Associate Director of CERP

Introduction

In 2009, an idea sparked that begged the question: what if there was a way to collect comparative data in service of rigorous program evaluation? That’s the question the Computing Research Association’s Committee on the Status of Women in Computing Research (CRA-W; now known as the CRA Committee on Widening Participation in Computing Research [CRA-WP]) and the Coalition to Diversity Computing (CDC) asked that ultimately started what is now known as the CRA Data Buddies Project. The Data Buddies Project has been running strong since 2010, and the conclusion of the fall 2020 survey cycle marked the 10^th year of the Data Buddies Survey: the flagship initiative of the project. This article dives into some of the history of the Data Buddies Project and the CRA Center for Evaluating the Research Pipeline (CERP) while also highlighting the project over the years. The article concludes with a look into how the project operates today.

Is it all just history?

In 2009, the Data Buddies Project was just an idea—one that was far reaching and would require skilled researchers to manage. The goal was to collect survey data from undergraduate and graduate students in computer science degree programs, which would then turn into a comparison group for CRA-W/CDC program participants. The outcome would be rigorous program evaluation that compared participants to a national sample of non-participants.

CRA-W/CDC leadership and CRA researchers worked together to design a pilot survey they called the Data Buddies Survey (DBS). They designed the survey to ask questions related to students’ sense of belonging to computing, their skillsets and interest in research, and their academic and professional aspirations. Academic departments across the United States would distribute the survey to their students during the fall and spring semesters: the fall semester to survey continuing students and the spring for graduating students.

After pilot testing the survey in the spring of 2010 with a small group of ten computing departments, DBS started strong with 54 academic partners by the spring of 2011 who agreed to work with the project for the 2011 and 2012 years. Although more than half of those departments were within doctoral degree granting institutions (52%), 28% of institutions were master’s granting and 20% were bachelor’s granting. In exchange for their time and recruitment efforts, these original 54 institutions were offered stipends and a free report for the department. The department report included summary tables that showed a given DBS school’s aggregate student responses alongside four comparison groups: top doctoral granting institutions, all other doctoral institutions, master’s institutions, and bachelor’s institutions.

Although researchers at the CRA were able to run the survey from 2010 to 2012, it soon became clear that the Data Buddies Project would require a full-time staff to manage the survey. With a new Broadening Participation in Computing (BPC) Alliance award (CNS-1246649) in hand, the CRA-W/CDC Alliance used some of those funds to start a new research and evaluation center called the CRA Center for Evaluating the Research Pipeline (CERP). By the fall of 2013, CERP was fully staffed with researchers with social science backgrounds who could take full responsibility of the survey, its data, and resulting reporting. The CERP team refined components of DBS, recruited more schools to participate, and improved the department report in ways that further helped participating departments understand their students’ responses.

With the help of the CRA-W/CDC Alliance funding, CERP developed a sustainable infrastructure for DBS that would enable the project to continue for years to come. Part of that sustainable infrastructure included submitting proposals to NSF to sustainably support the project. In 2014, CERP was awarded an NSF grant (DUE-1431112) to begin collecting longitudinal data alongside the foundational data collection procedures established in 2011. CERP has been sustaining the Data Buddies Project with additional funds including a large NSF IUSE grant (DUE-1821136) received in 2018.

DBS over the years

As one might expect, DBS has grown in many ways since 2010. This section showcases the departments and student responses that have contributed to the overall success of the Data Buddies Project.

DBS Partner Institutions

As mentioned earlier, the Data Buddies Project began with only ten institutions in 2010. Over a ten-year period (2010-2020), the number of participating institutions grew 1,350%! Out of the 234 computing departments that have been part of the project across the years, two stood the test of time and are currently the longest-standing partner institutions of Data Buddies: Columbia University and Duke University.

In Table 1 below, we showcase our long-standing partner institutions who joined in either 2010 or 2011 and have consistently participated in the project over the years except one or two years. In Figure 1, we also show the total number of DBS partner institutions from 2010-2020. Starting in 2015, CERP began highlighting “Elite” DBS partners who received a 20% or greater response rate. All active and elite DBS partners are included in Figure 1.

Table 1. Long-standing DBS partner institutions.

Institution	Department	Year Joined
Columbia University*	Computer Science	2010
Duke University*	Computer Science	2010
Tufts University	Computer Science	2010
Virginia Tech	Computer Science	2010
Colorado School of Mines	Electrical Engineering & Computer Science	2011
Harvey Mudd College	Computer Science	2011
Kean University	Computer Science	2011
Miami University-Oxford	Computer Science & Software Engineering	2011
New Mexico State University-Main Campus	Computer Science	2011
Radford University	Department of Information Technology	2011
Rochester Institute of Technology	Computer Science	2011
SUNY College-Plattsburgh	Computer Science	2011
Texas Southern University	Computer Science	2011
University of California-San Diego	Computer Science & Engineering	2011
University of Colorado-Boulder	Computer Science	2011
University of Hawaii-Hilo	Computer Science & Engineering	2011
University of Illinois-Springfield	Computer Science	2011
University of Maryland-Baltimore County	Information Systems	2011
University of Massachusetts-Amherst	Information & Computer Sciences	2011
University of Nebraska-Kearney	Computer Science & Information Technology	2011
University of Puget Sound	Mathematics & Computer Science	2011
Washington and Lee University	Computer Science	2011
Western Oregon University	Computer Science	2011

Note: (*) Longest-standing DBS Partner Institutions that joined in 2010 and never missed a year of data collection.

Figure 1. DBS Partner Institutions from 2010-2020.

Note: “Active Buddies” includes all DBS departments that actively participated in the survey each year. “Elite Buddies” includes active departments that received a 20% or greater response rate from students.

Spring DBS for graduating students

When DBS was first launched in 2010, it began as a spring semester survey for graduating computer science majors. A total of 386 senior undergraduates completed the inaugural survey across the ten partner institutions. From that point, both undergraduate and graduate students finishing their computing degree programs completed the spring DBS survey, which primarily collected data about their academic and professional aspirations.

To reduce the burden on departments distributing two surveys per year, CERP decided to retire the graduating student survey after the 2019 data collection wave. Unique measures collected from senior students were folded into the fall survey so that CERP could continue to learn about the career pathways of computing students in the field. Figure 2 below displays the responses collected through the spring survey from 2010-2019.

Figure 2. Responses to the spring DBS from 2010-2020.

Note: Data were not collected from graduate students in 2010. The survey was retired after data collection in 2019; as such, 2020 data were not collected.

Fall DBS

Now considered the flagship survey, the fall Data Buddies Survey has been actively collecting data from undergraduates and graduate students since 2011. Over time, the survey has grown to accommodate different types of students and even non-students, including alumni and professionals. One major change to broaden the survey was implemented in 2018 so that the measures included in the survey could be completed by both computing majors and non-majors – an important distinction that enabled departments to learn about the experiences of non-majors completing computing-related coursework. Although that option was somewhat available in prior years, the specific wording of many of the survey questions were difficult to complete by non-majors.

In 2019, another set of major changes were implemented to the surveys. Students in non-degree pathways such as certificate programs were provided a specialized pathway through the survey; in prior years, these non-degree students were not often eligible for the survey. The CERP team also began a pilot Data Buddies Survey for alumni and professionals – a non-student survey pathway to accommodate those who had already graduated and had entered the workforce. This additional pathway also opened doors for the CERP team to track more CRA-WP program participants. For example, participants of the CRA-WP Career Mentoring Workshops were now eligible to complete DBS.

Of course, these additional pathways meant more individuals were eligible to complete the survey. When we look at responses over time (Figure 3), we can see a jump in the number of responses to the survey in later years. These changes were important to make however, as the goal of DBS data is to provide as much information back to the community as possible about the state of higher education in computing, with a particular lens pointed toward understanding the experiences of students (and now individuals) in computing and technology fields. The integration of non-traditional pathways through computing into the survey was in service of this goal.

One important question the CERP team is often asked relates to the number of responses to the survey relative to the number of departments participating in the project. Figure 4 displays the average number of responses from undergraduate and graduate students per institution over time. As shown in Figure 4, the average number of responses per institution for undergraduate students steadily increased over time, apart from downward trends from 2011-2013 and in 2019. The average number of responses per institution for graduate students stayed steadier over time, though saw a small increase over time from 2015-2018.

Figure 3. Undergraduate and graduate student responses to fall DBS from 2010-2020.

Note: Data were not collected from undergraduate and graduate students in 2010.

Figure 4. Average number of responses from undergraduate and graduate students per DBS partner institution from 2010-2020.

Note: Data were not collected from undergraduate and graduate students in 2010. Average number of responses per year were calculated by dividing the total number of student responses by the total number of DBS partner institutions. Calculations for graduate student averages did not include Bachelor’s-only institutions.

DBS today

Today, DBS is running stronger than ever. DBS departments can request customized reporting with CERP’s new adaptive reporting infrastructure. CERP has also started reporting DBS data at an aggregated level through the new DBS Annual Report. As a next step, CERP will start reporting DBS measures longitudinally in an interactive report like the one available on an annual basis.

In addition to the various types of reports, CERP uses DBS data to publish infographics with research findings through Computing Research News (see Figure 5). These infographics typically feature an important research question related to understanding the experiences of students in computing and technology, often with a lens focused toward students from populations considered underrepresented in computing or historically marginalized (i.e., women; people who are Black/African American, Hispanic/Latinx, Indigenous and First Nations, Native Americans, Alaska Natives, Native Hawaiians, and Pacific Islanders; persons with disabilities; persons from low-income backgrounds; first generation college students; LGBTQIA+ individuals; and veterans).

Figure 5. Example of CERP Infographics in Computing Research News.

Finally, CERP recently overhauled the Data Buddies webpage to include more details about the project, how to get involved, and a comprehensive set of frequently asked questions. Departments can request copies of the DBS survey, and researchers can request a de-identified, aggregated dataset. CERP announced this update through a blog post featured on the brand new CERP Bulletin, the go-to resource for keeping up with CERP’s activities.

Is your department on the list of Data Buddies? If not, we encourage you to join today! To join the 2021 survey cycle, new departments should submit a volunteer form no later than November 1, 2021. This allows the CERP team to have time to engage with your Institutional Review Board (IRB), ensuring the project can move forward prior to the close of the survey.

About the author
Heather Wright serves as the Associate Director of the Center for Evaluating the Research Pipeline (CERP). In her role, Wright leads CERP’s evaluation efforts aimed at broadening participation in computing research and education. Wright also coordinates CERP’s various research initiatives and supports the continued sustainability of the center. She has been with CERP and CRA since 2013.

---

This analysis is brought to you by the CRA’s Center for Evaluating the Research Pipeline (CERP). CERP provides social science research and comparative evaluation for the computing community. Subscribe to the CERP newsletter here. Volunteer for Data Buddies by signing up here.

The Data Buddies Project is currently supported through National Science Foundation (NSF) awards CNS-1840724, CNS-2036717, DUE-1821136, sub-awards and contracts, and direct CRA contributions. Previous NSF awards that supported DBS include CNS-1246649 and DUE-1431112. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.