As the number of residency applications continues to rise with each passing match season (now 60+ each for US grads and 150+ for International Medical Graduates each year and growing), the residency application and interview process becomes more competitive. As a result, residency program leadership is burdened with the task of reviewing and screening more applications, now numbering in the thousands for certain programs managing the residency interview process.
This process is an incredibly heavy burden on program directors and program coordinators especially. The problem has become so pronounced that it has been written about extensively in the medical literature—including the Journal of Graduate Medical Education.
One paper dove into the logistics of reviewing thousands of applications. In 2016, Berger et. al described in the abovementioned journal the time costs of reviewing 1000 applications at 10 minutes per application on a PD. The result was 167 total review hours, or about 10 weeks worth of work (assuming 8 hours of review per day, for 2 non-clinical days per week).
This is, of course, nearly an impossible amount of review for a single program director. And this is only for reviewing applications.
When the goal is to recruit the “best” candidates to your residency program, interview time costs and ultimately successful resident matches take precedence and only add complexity to the equation. As there are limited metrics to determine optimal recruitment strategies, this post outlines why nuances in the interview invitation strategy employed by your program, may ultimately affect your match list more than initially anticipated.
One: The candidate pool is limited.
Around 35,000 candidates match each year into available residency positions in the US across all specialties each year. However, what this essentially means is the number of actual candidates that might attend your program may be limited to only a few hundred medical students, for smaller specialties like integrated plastic surgery (234 applicants in 2018-2019), versus larger specialties, like categorical internal medicine (12,527 applicants in 2018-2019).
The National Residency Matching Program (NRMP) provides a very useful and detailed data report each year. The 2018-2019 statistics may be found here.
The Electronic Residency Application Service (ERAS) also has helpful statistics which may be found here.
Two: Programs do not recruit in a vacuum.
With a limited number of candidates accessible to all programs, it is also worth noting that all programs in a specialty are essentially competing against all others for the same finite pool of candidates. What happens at one program in terms of recruitment, certainly affects other programs within each specialty.
Three: What happens at the “top” programs in a specialty ultimately affects what happens at all the other programs in that specialty.
While the metrics regarding which program in the country is the “best” are debatable, perceived reputation and prestige are main drivers for applicants as they enter the match to find the “best program” at which to train.
While these are not always defined by concrete or tangible metrics, many candidates find themselves on the Student Doctor Network, the subreddit for medical students, or alternative online social media platforms to learn about program reputation, competitiveness, and other soft metrics. Similarly, other candidates find themselves on the American Medical Association’s (AMA) Residency Data FREIDA Onlineor Doximity’s Residency Navigator, which ranks perceived reputation of programs based on the methodology listed here.
These tools have been studied extensively in the literature, with the Doximity Residency Navigator receiving a significant amount of study (see the Western Journal of Emergency Medicine and this studyby the Journal of Surgical Medicine).
One paper in the Mayo Clinic Proceedings reported that:
“Of 1241 applicants who reviewed Doximity reputation rankings, 977 (79%) reported that the rankings influenced their application, interview acceptance, or match list rankings of residency programs. Despite the significant impact of Doximity reputation rankings, 699 of 1240 applicants (56%) believed that the reputation rankings were not accurate or were only slightly accurate. Furthermore, only 314 of 1186 applicants (27%) believed that DRN helped them match successfully during the NRMP 2017 Main Residency Match.”
Thus, reputation appears to be a large driver of quality for candidates. And while the metric isn’t perfect (and at times significantly inaccurate) it tends to be the best information currently available for candidates. Specialty-specific organizations are attempting to find more objective metrics for programs, alongside our own efforts to use data to optimize and streamline residency program recruitment. However, it is important for program leadership to recognize that this is an unavoidable part of the residency application process.
As such, some programs are highly sought after while others are less so. And while this is likely completely intuitive to all involved in this process, what happens at the most highly sought after programs dictates what happens in the remaining programs in the specialty.
Team Thalamus has spent extensive effort studying residency interview and match data over the past half-decade across all major specialties. What we have learned is that the strongest candidates (going by USMLE Step 1 score and AOA status) tend to be accepted to the strongest programs by reputation. And while the USMLE Step 1 score is a poor predictor of both success in residency or board certification (which is another topic in itself), it is currently one of the main drivers of the residency application process.
Put this all together, and long story short: the top programs in a specialty get the top candidates determined by the most widely-used metrics. Then, a domino effect occurs—as program reputation drops, candidate metrics drop (with a significant correlation).
And this means that if your program tries to recruit to aggressively, for candidates that are out of reach, your program will be competing with programs of higher perceived reputation, and as a result, candidates your program is unlikely to recruit, further shrinking your available candidate pool of potential matches, ensuring higher cancellation rates and lowering rank list yield/efficiency.
The even shorter point is: “TARGET CANDIDATES LIKELY TO WANT TO MATCH AT YOUR PROGRAM DUE TO YOUR PERCEIVED STRENGTH OF REPUTATION BY CANDIDATES.”
Four: The Match is like the NBA or NFL draft… except VERY DIFFERENT.
If we look at professional sports league drafts, each team takes turns selecting players from a pool of potential athletes in a previously determined order one at a time. For the match, the pool is limited, but everyone picks at once by proxy through their rank order lists as part of the National Resident Matching Program (NRMP). Whereas in the draft it is made known to teams when a player is selected and therefore unavailable, this information is not made available as the NRMP algorithm is run. What this means is that there is significant potential for top candidates on your rank list to have been selected by other programs, well before your program even had the chance of matching them.
Similarly, in a sports draft, if a player is selected by a team, they become part of that team (unless they are traded or have contractual issues). While a successful NRMP match is binding between candidate and program, the NRMP algorithm itself is “applicant-proposing,” meaning that the applicant rank order list ultimately dictates which program at which a candidate matches. For programs, this means that not only does your program’s reputation need to be strong, but your goal is to maximize the number of candidates who rank your program highly to ensure that you are recruiting the “best” residents.
Five: A subset of your candidates will have a 0-5% chance of matching with your program.
Candidates are applying to way too many programs. However, they can ultimately only attend one program. This means that every other program that they interview at has essentially held an unsuccessful interview with that candidate. It also means that within your program’s group of candidates selected to interview, you want to limit the number of candidates who will likely not attend your program, while maximizing the number of candidates who are very likely to attend your program.
Who are those candidates? That’s the $64,000 question! At times it’s difficult to predict: academic profile, prior recruitment history/experiences/successes, candidate and medical school geography may all play a role. While we are working on our own algorithms at Thalamus to help assist residency programs in this regard, this is an important concept to be aware of and note while managing the residency interview process.
Six: How you conduct your interview season can ultimately dictate match success.
To summarize: The candidate pool is limited. Programs do not recruit in a vacuum. Top programs dictate what happens in a specialty in terms of overall match results. Reputation plays a major role in determining such. Candidate preference dictates program match success. Filtering out candidates unlikely to attend your program is a major factor to consider in managing the residency interview process.
What this all means in the greater context is that 1) not all programs are created equal, 2) not all candidate preferences are created equal, and 3) recognizing these discrepancies can ultimately determine success.
Indeed, what may appear as minor points during interview season, may affect all of the above. THE NUANCES MATTER.
When your program decides to send out invites (early vs. late), how many invites your program sends out, and your program’s geographical location can all have a major impact on cancellation and interview completion rates throughout the match season.
However, what may not be recognized is that these are not absolute values, but rather determined on what the top programs in your specialty are doing as well as your program’s closest competitors. Which of them have sent out invites? What days are they interviewing on?
Residency applicants are interviewing at as many programs as they need to feel comfortable to ensure they gave the best effort in the match. This changes for each candidate depending on number of interviews and time of year. You may be a top program in the northeast, which has trouble recruiting candidates from California. Others are stellar training programs that don’t have as strong a perceived reputation.
Rather than consider that your program is losing candidates to competitors, the first step we recommend is to consider “will this candidate actually attend our program?” While outliers exist everywhere (including for this born and bred New Yorker who trained in California), managing the residency interview process from the macro and micro perspective, and taking these lessons into account will give any program greater insight into their recruitment processes.
Sticking with the sports metaphors, programs can optimize their match outcomes using the popular baseball concept of Moneyball. Specifically “Using statistical analysis, small-market teams can compete by buying assets that are undervalued by other teams and selling ones that are overvalued by other teams.”
Statistics and data matter. The nuances matter. And using all available knowledge for your program will ultimately result in a more successful match process.
Navigate it with Thalamus
Want to learn more about how Thalamus can help manage the residency interview process? We are not only the most-preferred interview scheduling software, but a data-driven recruitment engine that exists to help residency program optimize their recruitment through evidence-based, statistical analysis. Need assistance with your recruitment? Request a demo with us to discover how Thalamus can streamline your recruitment process today!