This 5-part blog series, The Ultimate Guide to Preference Signaling for Medical Residency for Applicants and Programs 2022-2023 includes:
- Part 1: Introduces the history of preference signaling.
- Part 2: Describes its introduction to GME.
- Part 3: Discusses outcomes to date.
- Part 4: Provides advice for applicants and programs.
- Part 5: Describes how Thalamus will support this initiative.
This is Part 3: How Preference Signaling Affected 2021-2022 Application, Interview and Match Statistics
How has Preference Signaling affected application numbers and Match statistics?
Given the broader use of preference signaling for the 2021-2022 application season following the ENT pilot, our analysis will focus on the outcomes observed during this season in the four specialties that utilized ERAS and the NRMP Match, that also used preference signals: Otolaryngology, Internal Medicine, General Surgery and Dermatology. Important Note: While preference signaling was widely used in Otolaryngology, a smaller percentage of programs in Internal Medicine, General Surgery and Dermatology programs participated in the initial ERAS pilot, so increased specialty-wide effects may be observed with broader adoption.
Overall, across all specialties, the 2021-2022 ERAS application season saw increases in US MD and DO applicants and applications. There were also decreases in corresponding IMG populations in terms of applicants and applications (Partly due to visa challenges from the pandemic amongst others).
These trends were consistent across all specialties, regardless of whether they used preference signaling or not. Specific data for Otolaryngology, Internal Medicine, General Surgery and Dermatology are also shown here for comparison.
When comparing match rates over the last two years across the four preference signaling specialties, no substantial differences were observed. Dermatology and General Surgery had slightly fewer unfilled positions, internal medicine had more unfilled positions, and Otolaryngology filled all positions per usual. Non-signaling specialties saw the largest increases in unfilled positions including Emergency Medicine, Family Medicine and Pediatrics. No specialties matched at significantly higher rates than usual.
Further, there was no significant difference in length of applicant rank list in the signaling specialties between 2021 and 2022. There was no significant difference between signaling and non-signaling specialties. Emergency Medicine had the largest jump in applicant rank order list (ROL) length, likely due to a large decrease in the number of applicants applying into the specialty and significant increase in the number of available positions in the match (meaning more interviews were completed per applicant).
How has Preference Signaling affected interview allocations?
At Thalamus, we often use an interview metric we refer to as mean percent interview overlap to assess the competitiveness of the interview market in a specialty. Previously, we have used this metric to alleviate concerns about the possibility of an impending “Match Crisis” due to perceived “interview hoarding.” However this metric is also useful in assessing changes in interview allocation amongst programs.
Mean percent interview overlap is calculated by pairing interviewee lists of each program in a specialty and determining the mean overlap (cooccurrence of interviewees in each pair-program’s list). For example, if a specialty had a mean percent interview overlap of 10%, this would mean that if you picked any two programs in a specialty at random (which use Thalamus) and compared their interviewee lists, you would expect 10% of their interviewees to be shared.
For 2021-2022, amongst the four preference signaling specialties, we saw small but significant decreases in overlap in Dermatology and Surgery, a small and non-significant decrease in overlap in Internal Medicine and essentially no change in overlap in Otolaryngology. Specialties that did not use preference signaling varied by nonsignificant increases/decreases or no change. From these results and additional analysis, we might infer that preference signaling had some impact on decreasing mean percent interview overlap by redistributing interviews from applicants in the 2nd quartile (by interview allocation) to the 3rd and 4th quartiles.
We also look at the distribution of the mean numbers of interviews received by applicants. Here, for 2021-2022, we saw no distinct difference across any specialty compared to the prior year, nor did we find any significant difference between signaling and non-signaling specialties. All four signaling specialties show near identical means, as well as similar patterns of exponential decay. The following graph for Internal Medicine is shown as an example:
In summary, no effect was observed in terms of global application or match rates across any of the signaling specialties. There was a slight decrease in mean percentage interview overlap in Dermatology and General Surgery, but no change in the distribution of interviews received by applicants in these specialties or any of the other preference signaling specialties.
The preliminary Otolaryngology data showing the likelihood of interview invitation increasing across applicant groups to programs in which a preference signal was sent is a positive. More research needs to be done to look at prior recruitment seasons to assess: 1) Would these have been programs where applicants would have historically received interviews anyway? 2) How does this affect MD vs. DO vs. IMG (both US and Non-US Citizen) applicant populations separately? 3) How does this affect DEI across applicant groups, including self-identified ethnicity, etc.? 4) Is preference signaling equitable across these groups?
A big question remaining is precisely what values of what measures would preference signaling need to achieve for the intervention to be considered a success?
And moreover, how should applicants and programs approach preference signaling? Find out in Part 4!