Taulbee Report 2024

By Heather M. Wright, Associate Director of CERP

The CRA Center for Evaluating the Research Pipeline (CERP) is growing as one of the largest resources of individual-level survey data on undergraduate and graduate students in computing degree programs. Through the Data Buddies Project, also known as the Data Buddies Survey or just Data Buddies, CERP has been collecting annual data from students since 2011. Additional funding granted from the National Science Foundation to CRA and CERP in 2014 enabled CERP to begin collecting focused longitudinal data, so that each year, a new cohort of students are tracked every year of their academic degree program until graduation.

As a result, CERP has a wealth of data that can provide valuable insights to students’ trajectories into and out of academic computing programs at multiple levels. Between 2013 and 2017, CERP collected survey data from more than 40,000 undergraduate students and 14,000 graduate students in the United States and Canada. Data collected through the project are then provided back to the community through multiple initiatives, including the annual Data Buddies Department Report and CERP’s Data Visualization website.

Departments that volunteer to send CERP’s online survey link to students receive their annual report during the late spring semester. The report summarizes survey data specific to the department and delivers comparative data to offer context for survey results. A list of volunteer departments can be found here. In addition, CERP updates its Data Visualization website with current survey data for public use. Through the interactive website, visitors can select data they wish to see in a visualized format.

So how did Data Buddies finish for the 2018 reporting year? CERP collected approximately 10,500 survey responses from undergraduate students and 4,000 responses from graduate students in computing-related degree programs. Of those, 18% of undergraduates and 34% of graduate students are part of CERP’s longitudinal data collection efforts to track student engagement and persistence in the field over time.

Because students who are tracked over time may no longer be a student in-between surveys, CERP also collected data from approximately 1,500 non-students in an effort to understand their current status and whether or not they persisted in the field. This additional way of tracking students over time will enable CERP to better connect the pathways of students who make transitions between higher education and their careers.

Increased Insights on Students’ Experiences
While the Data Buddies Survey had been tweaked over the years, never before had it seen quite as large of a change as it did for the fall 2018 data collection year. In an effort to provide the computing community with data most aligned with evolving department needs, CERP collected feedback from volunteer departments between the spring of 2016 and 2018. Using that feedback, CERP overhauled the 2018 survey in a way that provided a balance between consistency for longitudinal data collection and improved insights to many questions often asked of educational research data in computing.

New this year, departments will learn more about their students’ experiences as they relate to the department culture, resources, and coursework. They will also learn about their students’ background in computing (e.g., took computer science AP courses or learned a programming language) and mathematics prior to entering college. Students who completed the survey also rated their research ability and proficiency in a number of skills known for success as both a student and future professional in the field, whether in academia, industry, or government.

Not only are these data valuable for academic departments interested in learning more about their student body, these data are essential for understanding which psychological, sociological, and environmental factors impact students who are considered underrepresented in computing. Ultimately, CERP’s goal is to use Data Buddies data for rigorous statistical analysis and reporting in an effort to provide evidence-based research findings and best practices related to the broadening participation of students in the field. It is CERP’s goal to also use these data as a way to understand how best to retain students in computing once they enter their academic programs.

Improved Program Evaluation
With these new insights offered by Data Buddies, CERP is also afforded better grounding for program evaluation. Program evaluation is a central part of serving the computing community. At present, CERP conducts evaluation for a large number of programs led by internal CRA committees, external organizations, and academic institutions. It is CERP’s goal that new survey measures collected through Data Buddies will provide better resources for assessing the efficacy of various programs.

CRA-W, which is the CRA committee responsible for the creation of CERP and Data Buddies, is CERP’s largest client for program evaluation. CERP evaluates the DREU and CREU programs, Grad Cohort for Women, Discipline-Specific Workshops, Grace Hopper Celebration Research Scholars program, and the Virtual Undergraduate Town Hall series. Some of CERP’s work on CRA-W programs can be found on its website or monthly infographics published in Computing Research News.

Developing New Resources
In an effort to serve the community even further, CERP is using the opportunity of new data to create additional resources. In 2018, CERP received new National Science Foundation funding to develop evidence-based best practices for broadening participation in computing. CERP is partnering with Colleen Lewis, an associate professor of computer science at Harvey Mudd College, to collect institutional-level data and work directly with departments to harness their department report, among other resources, to increase diversity in the field. Evidence-based best practices developed from this new work with Lewis will be made available through CSTeachingTips, Computing Research News, CERP’s website, and publications and presentations.

CERP also has in the pipeline a new annual report that will use aggregated Data Buddies data to provide overall results from the survey. It is CERP’s hope that this new annual report will become a valuable resource for both researchers and practitioners seeking data on computing education research.

Get Involved and Influence Change
The Data Buddies Project has officially concluded data collection for the 2018 year, but CERP is recruiting new departments for future years to join its initiative to collect large-scale and representative data from undergraduate and graduate students.

Joining the project is easy and free. If you belong to a department serving computer science, computer engineering, information technology, or any other degree program pertaining to computing and technology, sign up by using our mobile-friendly form. Your department is not required to be a CRA member to participate in Data Buddies.

About the Author
Heather Wright serves as Associate Director of CERP at the Computing Research Association. She leads program evaluation and supports the center’s research efforts to help broaden participation in computing fields.

Acknowledgment
This article was supported by the following National Science Foundation grants, awarded to the Computing Research Association: CNS-1246649, DUE-1431112, and DUE-1821136. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author’s and do not necessarily reflect the views of the National Science Foundation.