Undergrad Explores Research in Neural Differential Equations

This Q&A highlight features Winnie Xu, a Finalist in the 2022 CRA Outstanding Undergraduate Researchers award program. Winnie graduated from the University of Toronto and is now a Student Researcher at Facebook AI Research (Meta AI). Some of her recent work in the generative learning space includes training generalist multi-game agents, a new model inspired by fractal compression, and a probabilistic framework for composing large language model generations. This interview has been edited for length and clarity.

What brought you to computing research?
I started as a pre-med student but gave myself the chance to explore research directions broadly. I recall being particularly excited by the rapid advancements in genomics, which led me to read into big data analytics and land my first computational biology position. After taking an introduction to programming course, I wrote my first few lines of Python code to help automate our biological data analysis pipeline. Gradually, my analysis code became more complex, handling multiple genome data files and running clustering algorithms to make more sense of them. As I learned the tools I was using were based on methods in statistics and mathematics, I became interested in machine learning (ML) and computation. Somehow, I had the right intuition, a rapid change of heart, and the courage to transfer into Computer Science—the rest is history.

How did you connect with your undergraduate research advisor?
I wanted to get more proficient in ML. It combines many areas in mathematics, statistics, and algorithmic reasoning which are all exciting to me. I cold-emailed a bunch of faculty members at the Vector Institute, essentially asking to be advised on a project I had in mind at the time. Having no experience in such topics, I self-studied and overloaded my courses to complete the Artificial Intelligence Specialist requisites sooner. There was one professor who was willing to give me a chance: I ended up completing my undergraduate thesis project with Prof. David Duvenaud.

Can you tell us about your project?
My research topic was heavily influenced by Prof. Duvenaud’s recent work on Neural Ordinary Differential Equations (NODEs). That work had gained a lot of attention and it was an exciting technical challenge to work in an area of mathematics that I was not familiar with. You may be aware of neural networks, or deep learning, which is a popular, modern method of modeling phenomena from real world data. Differential equations are a much older approach to modeling real-world phenomena. Neural Differential Equations combine aspects of both. In my project, we improved upon existing architectures for Neural Differential Equations. For example, we improved the calibration (meaning high confidence predictions are predicted more often) and robustness (meaning the model performs well even when input data is slightly corrupted, as an image might be corrupted by snow on the lens). This paper was featured as a spotlight talk at the NeurIPS 2021 Bayesian Deep Learning Workshop and later accepted to the Artificial Intelligence and Statistics Conference (AISTATS) 2021.

What challenges did you encounter when first getting started in research?
My desire to explore different research directions presented some challenges. Each time I’ve switched topics, I’ve had to quickly ramp up on the relevant literature and tools while working with people who were already experts. I’d be lying if I said I wasn’t consumed by imposter syndrome despite being adamant about gaining new opportunities. I’ve also found research to often be a lonely pursuit, especially in ML, compared to my previous experiences in wet labs. It has become apparent to me that the ability to overcome obstacles and ask the appropriate questions are essential skills.

Have any of your outside interests had any interplay with your research experience?
In my junior year, research became my top priority as I took a break from school during COVID. I was making a lot of research progress, so I continued as a visiting researcher part-time in industry while completing the remainder of my degree. When I could, I traveled to collaborate with other research groups. Balancing school and research was challenging, and my grades sometimes took a hit with different travel and conference deadlines, but I think this was a reasonable sacrifice and net positive.

Have you found that any of your personal identities affected your research experience?
As a woman and racial minority in America, I am often outnumbered by colleagues and advisors who are men. Until recently, I’d never had a woman manager. Colleagues have not always taken me seriously and have assumed that because I am younger, I am immature. Over time, I’ve learned to present technical maturity and professionalism to prevent these assumptions.

Do you have any advice for other students looking to get into research?
Find an area that really excites you and try to read some of the key papers in the area. Then, reach out to professors to get your hands dirty. I have always erred on the side of passion and aggression, so even if you get a ‘no’ initially, I would learn to take rejection not as a ‘no’ but as a ‘not yet’.

— Edited by Yasra Chandio and Nadia Ady