Explaining COVID’s Impact on the 2020-2021 Virtual Recruitment Season and NRMP Match Outcomes

The 2020-2021 medical residency recruitment and match season was unique. Like most processes affected by COVID-19, this recruitment season was mandated to be entirely virtual. While some industries easily transitioned to the online space, the previously ubiquitous practice of in-person interviews in medicine has always been considered necessary, bordering on sacred.  While interviewing virtually has likely benefits, the loss of the experience of visiting an institution in-person, in a recruitment process that ends with a matching algorithm, which in turn dictates where applicants will spend the next 3-7 years of their lives, is understandably daunting.

Further adding to the stress of both applicants and programs, the medical residency recruitment process is competitive, expensive and time-consuming. Students report taking up to $25,000 dollars in additional loans to pay for application fees as well as airfare and other travel costs. They then interview and attempt to match.  While the match rate for domestic MD graduates usually falls in the mid-90%s, securing a residency spot is somewhat more difficult for DO graduates. It is even more challenging for students graduating medical school outside of the United States, known as international medical graduates (IMGs), where Match rates may fall to 50 or 60%.

For residency programs, sometimes as many as 1000 applicants are interviewed to fill ~20 residency positions.  When factoring in faculty and administrative time, the costs escalate on the order of $250,000 per program per year.  

Now, add a global pandemic and the first ever virtual recruitment season in graduate medical education (GME) history.  All who participated in this year’s recruitment season were in unique, unknown territory.  How would this virtual process affect an already tense, anxiety-driven process?

Setting the stage.

Early on in the pandemic, it was argued by many that virtual interviews would lead to increased “hoarding” of interviews amongst “top” applicants, and significant interviewee overlap amongst programs, then leading to a larger-than-usual number of unmatched applicants and unfilled programs.  Announcements were put out by governing bodies and leadership organizations including the American College of Surgeons website postulating a possible “match crisis,” and correspondence from the American Association of Medical Colleges (AAMC) which included a second publication as late in the process as March 9^th, 2021. Concern was also expressed widely across GME specific online groups, listservs and communities including Twitter, Facebook, and Reddit. These publications and public discussions seemed to increase anxiety for the majority of stakeholders involved.

As a software platform that assists in residency interview management at over 2,000 ACGME accredited (and non-accredited) residency and fellowship programs, across more than 200 institutions and 100 specialties, and works with greater than 90% of Match applicants, Thalamus has collected several data points that could assist in providing insight into the virtual season.  As our CEO is a former applicant who completed his residency recruitment cycle during a truly calamitous season featuring Hurricane Sandy, our Co-Founder is a former Anesthesiology Residency Program Director and now Associate Designated Institutional Official (DIO) at her institution, and several of our team members are former program coordinators and GME administrators, we are passionate about producing data-driven solutions to simplify and streamline the recruitment process for all parties. As a result of our collective experiences, we felt compelled to try to understand what was occurring in December of 2020 with our blog post, “2020 Residency Recruitment Crisis? Data Says, Not So Much,” and provide reliable insights. 

This follow up piece incorporates additional details now that the interview season has completed, and the 2021 NRMP match data has been released, for what appears to have been an historic Match. 

­­­­­

So what does this mean:  The match was historic in that the number of applicants entering the match and submitting ranks lists, the number of programs and positions available and filled, the total number of matched US MD, US DO, US Citizen IMG and Non-US Citizen IMG groups collectively reached all time highs.  The overall percent of applicants matched (including in each subgroup) decreased slightly, with the most significant drop in Non-US IMGs (down 6.5%), due in large part to the fact that the number of applicants entering the match grew at a far greater rate than the number of available positions has increased. Per the NRMP’s press release, “The unavailability of medical licensure examinations in the early stages of the pandemic coupled with permanent changes to the scoring and administration of those examinations by the end of 2020 created significant challenges for IMGs this year and likely contributed to the decline. Additionally, changes in clinical rotations may have affected match rates.”

But what about programs?

Our analysis included 9 of the largest residency specialties as presented below.  With regards to the NRMP Data for these specialties, there was no significant change in the number of programs and/or positions that went unfilled.  In some cases, several of the specialties faired marginally better.

Define the Mission.

With the outcomes data defined, our goal was then to analyze interview data from the recruitment season to develop a better understanding of what occurred.

Step 1:  Measure the Prior Recruitment Season’s Interview Data.

In our December 2020 blogpost, Thalamus reported that based on all available data, the virtual interview season showed no noteworthy difference in overlap of interviewees across residency programs in 9 major specialties (Internal Medicine, Family Medicine, Anesthesiology, Psychiatry, Obstetrics and Gynecology, Pediatrics, Surgery, Orthopaedic Surgery and Emergency Medicine). If an individual were to select any two residency programs in a specialty, on average the number of applicants who would have interviewed at both programs would be between 5 and 15 percent (although rates vary by specialty).  These rates showed no change from the 2019-2020 recruitment season.

Additionally, we reported that there were no appreciable changes in the distributions of total interview invitations received by applicants. And furthermore, the distribution of total interview invitations to residency programs was also consistent with prior years.

Long story short, programs were not all inviting the same small pool of applicants. 

While our sample size ranged between approximately 20-50 percent of the programs in each of the specialties we analyzed, and represented 34,000+ applicants, we additionally performed several checks to convince ourselves that the data were both generalizable (able to be applied to the entire population of programs) and reliable.

Step 2: Identify what implications this may have for the Match.

Our first analysis looked at invitations across residency programs and the number of interview invitations received by applicants at the middle of the recruitment season. At that time, we felt that it remained too early in the calendar to postulate exactly what would happen in the Match, given that we could not yet calculate the interview cancellation and completion rates. All that could be determined at that point was that this year, while interviews were conducted virtually, there was nearly an identical interview invitation distribution from the year prior.  Yes, there were several applicants receiving 50+ interview invitations, but this was a tiny minority of outlying applicants that had existed in prior years as well. The average applicant was in the single digits on average, which again was consistent with prior recruitment seasons.

Step 3: At the end of the season, compare the interview completion and cancellation rates.

As soon as the final residency interview in our system took place and the season closed out with the submission of rank lists on March 3^rd, 2021, we began analyzing the total interview completion and cancellation rate amongst programs, and our findings were as follows:

1. The number of interview invitations sent out by residency programs remained nearly identical to the 2019-2020 recruitment season, but the number of available interview spots increased across most programs. Programs likely made a conscious choice to interview more applicants given the virtual nature of the recruitment season, in what seems to be have been an intentional strategy to grow their number of candidates with completed interviews.
2. Overall, the average applicant completed 7.8% more interviews than last year. The increase in interview completion varies by specialty, ranging from approximately 5% to 10%.

• Programs on average completed 16.5% more interviews than last year. This increase also varies by specialty, ranging from approximately 15% to 25%.

• The average cancellation rate amongst applicants decreased from the prior season (uniformly across groups of applicants) likely due to the decreased opportunity costs of interviewing virtually. Indeed, it seems that, since interviewing from one’s home is significantly less expensive than interviewing in-person, applicants chose to complete more of their offered interviews than they had in the prior season. The uncertainty of the virtual interview season likely contributed to this as well.

• This decrease in the overall cancellation rate naturally skewed towards students who had more interviews, with the top 20% of applicants (by invitation count) completing roughly 10% more interviews than they had the year prior, but all groups in the remaining 80% of applicants also completed an average of 5-10% more interviews. We also saw significantly more unique applicants receiving an opportunity to interview.

• While candidates did hold onto more interviews, “interview hoarding,” even taken in the broadest of definitions, was not appreciably different than in the prior season.  In fact, across the entirety of the interview distribution, candidates were able to complete more interviews than in the season prior.

OVERALL FINDING:  So in sum:

1. The number of interview invitations stayed the same.
2. The number of interviews completed by both applicants and programs went up. 
3. The rate of interview cancellations decreased. 
4. And while the candidates receiving the top 20% of interview offers completed more interviews than other candidates, overall applicants and programs both completed more interviews, thereby lengthening rank lists and providing each greater opportunity to match.

While the slight decrease in match rates is due to a disproportionally larger number of applicants entering the match in comparison to the growth rates of the number of available residency positions, more candidates matched than ever before, because significantly more unique applicants received opportunities to interview.

And therefore, overall, the match rate held steady as it has had in recent years, driven by its Nobel Prize winning application of the stable marriage algorithm.  

Of course, there are several factors at play here including where applicants and programs enter preferences of where or whom they would like to match, respectively.  Similarly, given visa restrictions IMGs were particularly disadvantaged this year more than usual, which lead to their larger resultant drop in their match rate.  There are continued challenges here including increasing the number of positions of available training positions to match a continued acceleration and growth of the applicant pool, and data can help expand upon this work. 

Overall, the challenges of over-interviewing in GME was not greatly affected by the virtual interview process, and the greatly hypothesized “match crisis” appears to have been avoided.  Yet, this process shed significantly light on a continued systemic problem: Due to a lack of transparency, applicants continue to overapply, and residency programs continue to over-interview, creating a costly and anxiety-ridden process for all.  Data and technology can help change this for the future as COVID leaves its recognizable mark on the medical residency recruitment and match process.