CIFellows 2020 For the Record

< Back to CIFellows 2020 for the Record Landing Page

< Back to Appendices Landing Page

Appendix P: CIFellows 2020 Selection Process

July 13, 2020

Please note that we seek a holistic process for the benefit of the “union” of computing. Intellectual area is but one balancing criterion, albeit one that is convenient for obtaining parallelism among SLC members. Nevertheless, we should mitigate framing discussion as confederation of sub-areas. The Area diversity mechanism below is meant to support a policy falling in between strict federation of sub-areas and fully area-blind selection. 

Hard requirements:

  1. Max 2 going to an institution
  2. Max 2 coming from an institution

Goals: To strengthen the computing community by enhancing the academic career development of new PhDs affected by COVID-19 

  1. Highest merit (provided by reviewers) while considering holistic “balancing” goals
  2. As many BPC/CSEd awards as possible, up to 10, whose merit is similar to overall pool
  3. US citizens and permanent residents preferred (above their representation in application count)
  4. Gender, race, and ethnic diversity (above their representation in application count)
  5. Geographic diversity of pool as measured by going to EPSCOR state (at or above representation)
  6. Computing area diversity (awards should reasonably span areas and areas with a large number of submissions should be reasonably represented in the awards)

Individual diversity mechanism: label each application with a set of flags, tallied in the Flag Score, and added to the Technical Score

  1. Gender diversity (Female, non-binary) 
  2. Race/ethnicity diversity (Black, LatinX, Native American)
  3. US citizen and permanent resident
  4. Disabilities

Flags for identifying applicants that can be awarded with EWF money (up to 10), but NOT in total  score:

  1. CS education focus (CS Ed) in primary/secondary/ternary research area
  2. Broadening participation in computing (BPC)  in primary/secondary/ternary research area, and
  3. To/from minority serving or undergraduate institution (MSI)
  4. A flag for identifying applicant wanting to go to an EPSCoR state, to aid in seeing geographic diversity

Each reviewer scores application on 5 elements for max score of 20

Total score = (average of sum of reviewer scores) + 1 for each diversity flag 1 to 4 (up to 4 points), not 6-9

(Flags 6-8 affect the process below.)

Note: Selecting 59 Fellows from 550 applications from 426 applicants for an average success rate of 10.7% per application or 14% per applicant (but CSEd/BPC may be higher and others lower).

Area diversity mechanism for balancing computing broadly and per-area application pressure : 

For each research area beyond BPC and CSEd:

  • Seek a worthy candidate (or two) in all areas with worthy candidates and
  • Seek to fund at least 2/3 of each area’s proportion of applications, assuming worthy candidates.

Pre-Meeting Process: For each Primary Research Area done by corresponding SLC member(s):

  1. By COB Wednesday, email to Ann and Maddy ID and names of worthy “tentative accepts” for up to 3% of an area’s applications or 1 name. For EWF, name five applications (with MAX-1 to start) – this is not just top names by score.
  2. Outside of EWF by COB Wednesday, optionally send email to “save” any candidate in your bottom 50% of candidates, as the default will be to discard these candidates.
  3. Look more carefully at remaining candidates, e.g., for issues (great score divergence) or selection committee member COI. If COI inform Maddy and another selection committee member will handle if/when discussed. Concentrate on the applications that might be successful.
  4. Have (partially) ordered list of non-culled candidates using Total Score above. For maybe 15% of candidates (beyond the above 3%), prepare to succinctly present at the meeting with key strength (or two), key weakness (or two), and brief optional comments.

Using this information above we created tentative Accept/Reject lists and identified any max-two conflicts in the tentative Accepts. This was brought to the big selection meeting. 

Meeting Process: Done By Whole SLC.

  1. If COI, leave Zoom meeting, immediately re-join, and stay in the waiting room until readmitted.
  2. Consider all applications in CSEd and BPC area in total score order, tentatively taking up to 10 & revisiting after whole pool is considered. At least initially with MAX-1
  3. Consider all applications To/From minority serving in total score order
  4. Consider applicants from top Total Score down
  5. Checkpoint against all targets — this is an important step in the meeting where we will compare the characteristics of the tentative pool against our goals (research area coverage and demographic diversity)
  6. Adjust/proceed as needed to meet targets

** At each step in the process, check against max 2 rule

** Keep spreadsheet of tentative accepts with all relevant data including institution(s) 

Questions:

(1) How to handle institutions with lots of applicants and/or lots of mentors? (Actually these two tend to go together) When and how to consider these?

(2) How to manage two mentor institutions and options this provides to shuffle to meet max 2?