I have a confession. Making my program’s rank list is one of the least satisfying aspects of my work as a program director. After months combing through applications, meeting applicants in person, and worrying if we interviewed enough students to ensure that we fill all of our spots in the NRMP Match, we finally formulate a rank list and hedge our bets.
Hedging our Bets on our Program Rank List
In my opinion, hedging is a pretty accurate word to describe our process.
When a program director meets with his or her recruitment faculty to build a program rank list, their discussion is limited to qualifying each applicant in terms of conditions vs. exceptions. Conditions might be a high USMLE Step 1 (soon to be pass-fail) or Step 2 CK scores or medical school ranking. Exceptions can also involve the same criteria, albeit on the other tail of the normal distribution. Exceptions might also include other components including a red flag on interview day or perceived geographic constraints (an applicant living in Nebraska applying to a program located in Upstate, NY, for example).
Hedging our bets implies that we are somehow attempting to “game the match.” It implies that we are protecting ourselves against a loss (i.e. not filling) by ranking less qualified applicants higher on our rank list.
In truth, while I have never engaged in that type of manipulation, some programs do.
Therein lies the angst.
You see, program directors are people with feelings – just like applicants!
We also want to be loved. We want to feel needed. We want our programs to be highly desirable to candidates. And we want to train great residents who will become great doctors.
We covet highly qualified candidates and hope they will choose us. We also worry about applicants who end up in our program, but don’t really want to train with us.
In our case, when we finally sit down to review the 160-220 applicants we met in person, it is a bear to put our list of applicants in some sensible order. (Yes, I did interview 220 applicants for 16 spots during my first year as a Program Director… I was nervous!)
So, in the end, how does this get done?
Creating a Program Rank List: The Process
I can only describe how we do it. Here it goes.
Who does the ranking? Our interviewers include a mix of faculty, the program director, and the Chair. At the end of each interview day, these interviewers, and the chief resident tour guide of the day, and our program coordinator convene to debrief and score each candidate. Program coordinators are an awesome source of input, as they are usually the first and last line of communication with the applicants.
How do we score applicants? In some years, we scored on a scale from 1-10. Using that scale, almost everyone scored 8-10, and those few that we weren’t going to rank scored 7 or less.
To make our system more equitable, we then began scoring on a weighted average of categories involving IQ, EQ and even CQ (curiosity quotient). My eventual plan is to correlate these scores with graduate outcomes, but that is beyond the scope of this blog. Most recently, we have been scoring from 1-3 only, with 1 indicating a poor fit, 2 signifying this candidate would be fine, and 3 being the coveted “rank to match.”
What does “rank to match” mean? It means that if we have 16 spots to fill, that applicant is ranked somewhere from 1-16. However, if an applicant is placed at 25 on our rank list and really wants to match at our program, there is a pretty decent chance that applicant will match with us. We have NEVER filled before reaching candidate 25 on our list.
Post-Interview Communication Standards
I do have an approach for my post-interview communications.
When I am not sure about how we are going to rank someone, I might respond to an applicant’s email like this: Thanks for writing this thoughtful note. It was a pleasure meeting you, too. We look forward to keeping in touch as the season progresses.
Once we draft the rank list, the communications become less vague. There are some rules to govern these communications. We cannot ask the applicants how we fare, and we cannot ask them about their rank lists. Applicants may not ask us how we will rank them, either. However, both sides are allowed to TRUTHFULLY share their thoughts about each other. That is why we wait until we actually complete all interviews and make our list.
I would never tell an applicant that he or she is ranked to match unless I am 100% sure that this information is 100% true. Since I have met these criteria regarding certain applicants in the past, I have informed a very select group of applicants that they are “ranked to match.”
As an example, we had an applicant from our own medical school, who was Alpha Omega Alpha (AOA), wanted to stay in Rochester, and was planning her wedding. Thus, she really did not want to spend unnecessary time or money on interviewing. So, as soon as she disclosed this information to us, and we all agreed that we would LOVE to train her…we committed to sharing this information with her.
We care a great deal about this process and understand that applicants care a great deal too! Traditionally, we have communicated these feelings to the top third of our rank list with a note that read something like, “We would be honored to have you train in our program and plan to rank you [to match]//[in the top third of our list]/ /[in a place on our list where we have traditionally matched applicants who have elected to train in our program].”
We hope the wording of these emails is helpful and not confusing to the applicants. We want to provide some insight and transparency into our process. We hope that the candidates that write to us are doing the same.
We don’t write to everyone on our list. We often match very worthy applicants from lower down on our list, and we have overall been VERY pleased with our trainees.
So, why do we write to any of the applicants?
The Role of Post-Interview Communication in the Rank List Creation Process.
During our marathon rank-list making sessions, our first draft rank list is based on our post-interview debriefing scoring. Then, we move applicants around based on other “conditions” such as couples matching (Did ENT also like his or her spouse?) and geography (Do you think he or she will really move away from Oregon, when he or she has never lived to another state?), to name two examples.
Then, we review the post-interview communications. Was it personal? Did it seem sincere? What did he mean by “happy to train in Rochester?” How high is “rank highly on my list?” Is that code for they ranked us 5th out of 10 programs? Where is the love? Are we good enough?
“We have SIX commitment emails as of today,” exclaimed our associate program director. Do we rank those applicants in our top 16? If so, we might be able to report that we filled 6/16 in our top 16! We have heard that medical students like to know this information when evaluating programs.
Alternatively, do we confidently rank these SIX applicants somewhere in our top third, hoping that indeed they are being truthful, and that based on 25 years of history in our program, they will indeed match with us. Then, we can save higher spots for other “more coveted” candidates – like from Harvard or Yale (we have heard that medical students like to see that we matched Ivy League schooled applicants, although some of our strongest residents have trained at Caribbean medical schools and other traditionally “lower tiered” schools).
What it Comes Down To
What really happens, in our program, is that we rank applicants based on their merits. Merit, of course, is a somewhat subjective characterization. Merits are what we consider positive conditions:
- Strong academic performances as indicated by exam scores and class ranks.
- Emotionally intelligent as indicated by Gold Humanism Honor Society inductance, but also by your just being self aware.
- Curiosity as indicated by your research or scholarly pursuits, or just by the questions you ask us that do not seem scripted seemingly.
- Situational awareness as indicated by…well…not really anything in your application unless someone explicitly writes that about you. Sometimes we don’t find out you are NOT situationally aware until we see you in the OR.
As for exclusions? Seasonal dysthymic disorder, for one (it is winter here many months throughout the year…). If I mention that on the interview day and the candidate doesn’t laugh…that is usually an exclusion, too.
Thalamus is a fantastic tool for navigating the residency interview management process on the way towards building an NRMP rank order list. We publish these confessions to ensure that candidates have key insight into the process at large, so they can best position themselves to attend the residency program of their dreams.