Reading the results

Did the generator produce the applicants asked for?

A histogram of applicant strength (z-scores) in this run. When the applicants were generated, a line shows the counts the parameters ask for: the mixture of the applicant groups' normal distributions.

How to read it: Bars that follow the line mean the population is what the parameters describe. The summary gives the largest deviation in standard errors; above 4, the generator did not produce what was asked.

Note: Uploaded applicants have no line: their strengths are given, not drawn.

Did the generator produce the programs asked for?

A histogram of program quality (z-scores) in this run, with the counts the parameters ask for (the tiers' normal distributions, or one normal distribution without tiers) when the programs were generated.

How to read it: Bars that follow the line mean the programs are what the parameters describe.

Note: Uploaded programs have no line: their qualities are given, not drawn.

How are the positions spread over programs?

How many programs have each number of positions.

How to read it: A long right tail means a few large programs hold many of the positions. The key numbers above give the median and the range.

How well does each applicant know their true order?

For each applicant, the rank correlation (Spearman) between their true ordering of every program and their view of it before interviews.

How to read it: 1 is a perfect view; values near 0 mean the pre-interview view says little about the truth. The spread shows how unevenly information falls among applicants.

Note: Before interviews only: interviews reveal more, as the Applications and interviews tab shows.

How well does each program know its true order?

For each program, the rank correlation (Spearman) between its true ordering of every applicant and its view of it before interviews.

How to read it: 1 is a perfect view; values near 0 mean the pre-interview view says little about the truth.

Note: Before interviews only: interviews reveal more, as the Applications and interviews tab shows.

How accurately do applicants see programs?

True against observed utility for a sample of applicant–program pairs: before interviews, the true utility against the pre-interview view; after interviews, for a sample of interviews, the realised utility against the view after the interview.

How to read it: Points on the dashed line are seen exactly; the wider the cloud around it, the noisier the view. The summary gives each cloud's correlation.

Note: A sample of at most 1,500 points per series, the same on every visit (drawn from the run's seed).

How accurately do programs see applicants?

True against observed utility for a sample of program–applicant pairs, before interviews and (for a sample of interviews) after them.

How to read it: Points on the dashed line are seen exactly; the wider the cloud around it, the noisier the view. The summary gives each cloud's correlation.

Note: A sample of at most 1,500 points per series, the same on every visit (drawn from the run's seed).

Is demand concentrated on a few programs?

How many applicants rank each program first before interviews, most wanted first, with each program's number of positions as a mark.

How to read it: Bars far above their mark are programs many more applicants want than they can take. With many programs, zoom with the slider under the chart.

Note: First choices before interviews, not the final rank order lists.

How unequal is first-choice demand?

The Lorenz curve of first-choice demand: programs from the least to the most wanted, against their cumulative share of first choices.

How to read it: The dashed diagonal is equal demand; the further the curve sags below it, the more concentrated demand is. The Gini coefficient in the summary is twice the area between them (0 = equal, 1 = everyone wants the same program).

Where do applications drop out?

Every application through the stages: invited to interview or not, interviewed or not (declined, or over the applicant's interview cap), on the applicant's rank order list or not, matched or not.

How to read it: Each band's width is a number of applications; hover a band for its count and share. Grey marks drop-offs.

Note: It counts applications (applicant–program pairs), not applicants.

Which applicants get an interview and a match?

Every applicant counted once: whether they had at least one interview, and whether they matched. With Colour by strength on, each stage splits into five groups by strength percentile (fifths; applicants of equal strength stay in the same group), from the bottom 20% in red through the middle in grey to the top 20% in blue, and each drop-off shows which groups it holds, so the quality of the applicants can be followed through the stages.

How to read it: Each band's width is a number of applicants. Hover a stage or a band for its count and its share of all applicants (with the switch on, of its group) and to highlight its path; a band after the first stage also gives its share of the stage it leaves. No interview splits into never invited and invited without an interview. With the switch on, each drop-off's bar is split into the groups' colours in proportion, and the small bar beside its label shows the same mix from weaker to stronger; hover the drop-off for each group's count, its share of the drop-off and the share of the group it holds: a group that loses a larger share of its applicants does worse.

Note: Strength is the applicant's latent strength. The view all programs share is mostly strength, mixed with the applicant's attributes, and each program sees it only through its own noisy view. The funnel above counts applications instead: strong applicants hold many interviews but match only once.

Who gets the interviews?

Applicants by the number of interviews they had, from none upwards. With Colour by strength on, each bar is split into five groups by strength percentile, from the bottom 20% in red at its foot, through the middle in grey, to the top 20% in blue at its top.

How to read it: Each bar is a share of all applicants. A tall first bar is applicants no program interviewed; a tall last bar is applicants who reached the most interviews an applicant accepts (Interviews per applicant (max)). With the switch on, a bar's colours show who had that many: blue towards the right means the strongest applicants hold the most interviews. Hover a bar for the numbers. The summary says how concentrated the interviews are: the share held by the tenth of applicants with the most, and the Gini coefficient (0 = every applicant has the same number, 1 = one applicant has them all).

Note: It counts the interviews held, not the invitations: one an applicant declined, or could not fit under their limit, is left out.

Which choice did applicants match to?

Matched applicants by where they had ranked their program on their own rank order list: first choice, second, and so on, with every choice after the tenth counted together.

How to read it: Each bar is a share of the matched applicants; a tall first bar means most got their first choice. Hover a bar for the number of applicants.

Note: Choices are places on the list the applicant submitted, which holds only programs they interviewed at: a first choice here is not always the program they wanted most before interviews. Applicants who did not match are not shown.

Does a longer rank order list help?

Applicants with a rank order list, by the number of programs on it, and in each group the share who matched.

How to read it: Each bar is all the applicants whose list has that many programs; the green part matched. As in the real Match, the share usually climbs steeply over the first few programs ranked and then levels off. Hover a bar for the numbers.

Note: A longer list is a sign of a stronger application, not only a cause of matching: a list is as long as the applicant's interviews, and strong applicants get more of them. A bar that stands for a handful of applicants can be far off. Applicants without a list are left out.

Do stronger applicants match to better programs?

Applicants in five groups by strength percentile (the rows, the top 20% at the top) and where each group ended up (the columns): not matched, or matched to a program in each fifth of the programs by quality, from the bottom 20% to the top 20%. Each cell is its share of the row's applicants, darker for more: blue for matches, grey for not matched.

How to read it: Read along a row: it adds up to 100%. In a perfectly sorted market the dark cells run along the diagonal, from the weakest applicants, unmatched or at the lowest programs, to the strongest at the best programs; the more a row spreads out, the less strength decides where its applicants match. Sorting, in the summary, is the rank correlation between an applicant's strength and the quality of the program they matched to (close to 1 when the strongest applicants are at the best programs, in order; 0 = no relation). Hover a cell for its number of applicants.

Note: Applicants do not rank programs by quality alone: with tastes of their own (agreement below 1) a market that works well is still not perfectly sorted. Strength and quality are the latent values, which the other side sees only through its own noisy view. The fifths of programs hold equal numbers of programs, not of positions. Sorting cannot quite reach 1, because the applicants of one program share its quality: the fewer and larger the programs, the lower its ceiling (0.99 with 8 programs of 6 positions).

Which programs fill their positions?

The positions of the programs in each fifth by quality, from the bottom 20% of programs to the top 20%, and the share of them the match filled.

How to read it: Each bar is all the positions of its group; the green part is filled. A bar with grey at the top has unfilled positions: hover it for the positions, how many were filled, and how many of the group's programs have an unfilled position.

Note: In the real Match, unfilled positions go to the scramble after the match (SOAP), which is not simulated.

Do stronger applicants match more often?

Applicants in ten groups by strength (deciles: 1 is the weakest tenth, 10 the strongest) and, in each, the share who matched, who submitted a rank order list but did not match, and who had no list (no interview, or no program they would rank).

How to read it: Each bar is all the applicants of its decile; the green part matched. Hover a bar for the numbers and for the match rate among applicants with a list, the rate NRMP reports use and the one in the table below.

Note: Strength is the applicant's latent strength, which each program sees only through its own noisy view.

How far did each of this applicant's applications get?

Every program the applicant applied to, on rings by how far the application got: applied only on the outer ring, then invited, interviewed and ranked, and the match in the middle.

How to read it: Hover a program for its name. The table below lists every application with its details.

Note: Ranked means the applicant ranked the program; whether the program ranked them is in the table.

How far did each application to this program get?

Every applicant who applied to the program, on rings by how far the application got: applied only on the outer ring, then invited, interviewed and ranked, and the matches in the middle.

How to read it: Hover an applicant for their name. The table below lists every application with its details.

Note: Ranked means the program ranked the applicant; whether the applicant ranked it is in the table.

Where did this applicant's applications drop out?

The applicant's applications through the stages: invited to interview or not, interviewed or not (declined, or over the applicant's interview cap), on the applicant's rank order list or not, matched or not.

How to read it: Each band's width is a number of applications; hover a band for its count and share. Grey marks drop-offs.

Note: Ranked means the applicant ranked the program.

Where did the applications to this program drop out?

The applications the program received, through the stages: invited to interview or not, interviewed or not (the applicant declined, or was over their interview cap), on the program's rank order list or not, matched or not. With Colour by strength on, each stage splits into five groups by the applicant's strength percentile among all applicants, in the colours of the applicants' flow (the bottom 20% in red, the middle in grey, the top 20% in blue), so you can see whom the program invites, ranks and matches.

How to read it: Each band's width is a number of applications; hover a band for its count and share, and to highlight its path. Grey marks drop-offs; with the switch on, each drop-off's bar is split into the groups' colours in proportion, with the same mix in the small bar beside its label. The table gives every stage by group.

Note: Ranked means the program ranked the applicant; an applicant it ranked may have matched elsewhere. The groups are fifths of all applicants, so a program that attracts strong applicants has more of its applications in the top groups.

In which run do weaker or stronger applicants match more often?

Applicants in ten groups by strength (deciles: 1 is the weakest tenth, 10 the strongest), and for each the share of its applicants who matched, in run A and in run B.

How to read it: Where one bar of a pair is taller, that run matched more of the decile's applicants. Blue is run A and orange run B, as the key at the top of the page says. Hover a pair for both runs' numbers.

Note: Each run's deciles are of its own applicants. Unless both runs have the same applicants, the same decile holds different people.

In which run do applicants get a higher choice?

Matched applicants by where they had ranked their program on their own rank order list, first choice, second and so on, as a share of each run's matched applicants.

How to read it: A taller first pair means more applicants matched to their first choice. Blue is run A and orange run B, as the key at the top of the page says. Hover a pair for both runs' numbers.

Note: Shares of matched applicants: a run that matched fewer applicants can still show a taller first bar.

In which run are the interviews spread more evenly?

Applicants by the number of interviews they had, from none upwards, as a share of each run's applicants.

How to read it: Compare the first pair (applicants with no interview) and the last (applicants at the most interviews an applicant accepts). Blue is run A and orange run B, as the key at the top of the page says. Hover a pair for both runs' numbers.

In which run do programs fill their positions?

The share of positions filled in each fifth of programs by quality, from the bottom 20% to the top 20%, in run A and in run B.

How to read it: A shorter bar is a group of programs with unfilled positions. Blue is run A and orange run B, as the key at the top of the page says. Hover a pair for both runs' numbers.

Note: Each run's fifths are of its own programs.