This 5-part blog series, The Ultimate Guide to Preference Signaling for Medical Residency for Applicants and Programs 2022-2023 includes:
- Part 1: Introduces the history of preference signaling.
- Part 2: Describes its introduction to GME.
- Part 3: Discusses outcomes to date.
- Part 4: Provides advice for applicants and programs.
- Part 5: Describes how Thalamus will support this initiative.
This is Part 2: The Introduction of Preference Signaling to GME
How Preference Signaling came to be in GME:
Preference signaling was proposed via an Orthopaedic Surgery publication in 2017 titled, “Not the Last Word: Want to Match in an Orthopaedic Surgery Residency? Send a Rose to the Program Director”. A comparable article followed in 2018, published in JAMA Otolaryngology Head and Neck Surgery, proposing a “Star System” for ENT residency programs. Next, a computer simulation study in ENT was published in 2019 in the Journal of Graduate Medical Education (JGME), concluding that preference signaling improved the numbers of interview invitations and also promoted holistic review. It was concluded that signaling should be considered as a candidate intervention to combat application inflation and over-interviewing.
How Preference Signaling has been used in GME to date:
Otolaryngology was the first specialty to implement preference signaling for the 2020-2021 residency application season. This was also the first fully virtual recruitment season due to the COVID-19 pandemic.
This process was managed through the Otolaryngology Program Directors Organization (OPDO) website. The preference signaling platform opened two weeks prior to the start of the opening of the Electronic Residency Application Service (ERAS) recruitment season. Each applicant was allocated five tokens to use as preference signals to programs. ENT residency programs then received lists of applicants that sent preference signals to their program on ERAS opening day.
With nearly all programs participating in the pilot this first year, ENT extended this pilot for the 2021-2022 residency application season. Urology implemented a similar program through the American Urologic Association and Internal Medicine, Surgery and Dermatology residences implemented preference signaling through optional participation as part of a novel ERAS supplemental application.
How has Preference Signaling impacted the residency recruitment process to date?
Limited outcomes data are available as of the publishing of this article. Only preliminary data are available, released by OPDO around Match Day (March) 2021 for the 2020-2021 ENT preference signaling pilot, a post on Twitter, and limited data from the 2021-2022 ERAS season.
The summary level data provided by the OPDO depicts the following for the 2020-2021 ENT application process:
- 632 applicants applied to ENT via ERAS.
- 558 unique applicants (88%) participated in ENT preference signaling.
- 2,764 preference signals were sent across 125 programs (of 129 programs in the Match).
- 559 unique applicants registered for the NRMP match in Otolaryngology.
In terms of distribution of preference signals, 8% of programs (10/125) received 51 or more signals. 60% of programs (74/125) received 20 or fewer signals. Overall, the distribution was non-normal, skewed and right-tailed. Overall, the top 30 programs in otolaryngology (top 24%) received over 52% of the preference signals.
For applicants, ~50% of signals were sent to programs in the same region as their home program.
Overall, applicants (and programs, although not shown) approved of preference signaling:
Finally, the interview offer rate increased amongst applicants across all four quartiles of competitiveness (1st quartile = most competitive applicants), when compared to interview rate offers for both non-signaled programs (i.e. programs an applicant applied to but did not send a preference signal) and closest non-signaled programs (i.e. the next program to which an applicant would have sent a preference signal if given a sixth signal to send to another program).
Overall, the preliminary data showed satisfaction from both applicants and programs. Preference signaling also appeared to affect the rate at which interview offers were received, inclined towards home geographic-regions (which support several previous geographic studies analyzing the match, as well as a bias in the proportion of signals sent to the most sought-after otolaryngology programs.
On a May 17th, 2022 Twitter Thread, some additional information was shared during a retreat of Med Ed leaders and stakeholders. The graph presented by ENT leadership showed that a preference signal increased interview offer rates above the overall average.
As for the ERAS signaling pilot, the AAMC have briefly reported on survey responses and focus group feedback in their May 2022 Supplemental ERAS Application Guide. Preference signals within ERAS are at times referred to as “program signals.” Programs reported using preference (or program) signals in the context of a “full application rather than in isolation” in the following ways:
- Over 80% of programs used preference signals amongst respondents, and of those programs using preference signals across specialties used them in the following ways:
- 88-96% as part of a holistic review to decide whom to interview.
- 84-94% as a tiebreaker to determine to whom to send an interview invitation.
- 44-65% to prepare for interviews or stimulate questions during the interview.
- Over 50% reported they were able to identify applicants whom they would have otherwise overlooked.
The initial outcomes data appear positive, though there is still much to study as preference signaling interventions progress and evolve within specialties and are added to new specialties.
So how did this affect application, interview and match statistics? Find out in Part 3!