The 13-point USMLE question quality rubric.

Ten flaws drawn directly from the NBME Item-Writing Guide, scored as pass or fail per item, plus three USMLE-specific gates. The ten flaws split into testwiseness (grammatical cues, logical cues, absolute terms, word repeats between stem and key, convergence/longest-option) and irrelevant difficulty (vague terms, non-homogeneous options, unfocused stem, true-false-hybrid options, filler templates like 'all of the above'). The three USMLE gates are clinical-vignette structure (demographics, presentation, finding, lead-in), two-step reasoning (the item asks the test-taker to recognize then act), and a source-anchored answer key. Score 13 binary checks per question. Zero flaws is exam-grade; one is acceptable; two or more means the item is not safe to study from cold.

The ten flaw categories are paraphrased verbatim from the NBME Item-Writing Guide (nbme.org/educators/item-writing-guide), which is the source NBME item writers train against and which the literature (Downing 2005, Tarrant 2006, Rush 2016) uses to score retrospective audits of real exam banks. The three USMLE-specific gates are the conventions every NBME-style item follows whether or not the guide names them explicitly: a vignette opens with demographics, the lead-in is at the end, the test asks the candidate to do something with the data (most likely diagnosis, next step, mechanism). Studyly uses this same set of thirteen checks as the in-flight gate inside its generator for medical-school decks.

Pull ten consecutive items from the tool's output, not cherry-picked. For each item, walk the 13 checks in order. Mark each as pass or fail. Sum the fails per item; a high-quality item has zero. Across the ten items, count how many had two or more fails; that number divided by ten is your unsafe-rate. A reputable commercial QBank should run an unsafe-rate near zero. The held-out three-document eval at studyly.io/quality reports a related composite: factual correctness, clarity, distractor quality, question-type coverage, scored on a 0/3/7/10 anchor scale and renormalized to 0-100 (Studyly 81.3, Unattle 78.0, Gauntlet 68.0, Turbolearn 57.8).

Because they break the item in opposite directions. Testwiseness flaws make the right answer cheaper than it should be: a savvy test-taker can pick correctly without knowing the content, by reading cues in the options (grammatical agreement, longest option, repeated keywords). Irrelevant difficulty flaws make the item harder than it should be for the wrong reasons: vague stem, non-parallel options, time wasted on reading rather than thinking. A single item can have both. The 23.5-point spread on the held-out eval is heavier on irrelevant-difficulty failures than on testwiseness, because AI tools tend to write fluently but unfocused.

The four-criterion rubric (factual correctness, clarity, distractor quality, question-type coverage) is a generic study-question rubric on a 0/3/7/10 anchor scale, scored per card. That rubric is at studyly.io/t/ai-study-question-quality-rubric. This page is USMLE-specific: binary pass/fail, anchored to NBME's named flaws and the clinical-vignette conventions Step 1 and Step 2 CK actually use. They are compatible. If you want a single composite score across tools, use the four-criterion rubric. If you want a defensible go/no-go on a single USMLE-style item, use this one. Both reach the same conclusion on most items.

Word repeats between the stem and the keyed answer. The model writes a vignette mentioning 'wedge-shaped pleural opacity,' then keys the answer to 'pulmonary infarct' while one of the distractors literally repeats 'wedge.' On the held-out audit this single flaw shows up in roughly one of every four ungrounded AI outputs. The runner-up is the convergence cue (longest option correct). Both are testwiseness flaws and both are easy to fix at generation time with a one-line rule. The harder failures are irrelevant-difficulty: a non-homogeneous option list (one diagnosis, one mechanism, one drug class) that no rule catches without semantic understanding.

Yes. The ten NBME flaws are format-agnostic and the three USMLE gates are stricter on Step 2 CK and Step 3 than on Step 1. Step 2 CK items use longer vignettes and more often ask 'most appropriate next step in management' instead of 'most likely diagnosis,' which raises the bar on the two-step-reasoning gate (the item must require both recognition and a decision). Step 3 vignettes have even more clinical context and an explicit setting cue (clinic vs ED vs inpatient). All thirteen checks still apply. The unsafe-rate on AI-generated Step 2 CK items is higher than on Step 1 because the vignette structure is harder to get right.

Each generated card runs through the thirteen-check rubric before it is allowed into the deck. Items failing any testwiseness flaw are regenerated with a deterministic rule (length-match distractors, strip absolutes, rotate any stem-key word repeats). Items failing an irrelevant-difficulty flaw or a USMLE-specific gate are either regenerated with a different prompt path or, if regeneration fails twice, dropped. The dropped-card rate on medical-school decks is around eight percent, which is the honest cost of in-flight gating. The leaderboard score is the result of running the same rubric post-hoc on cards that already passed the in-flight gate, so the two scores should and do correlate.

Usually yes, with caution. A single flaw rarely makes an item dangerous; it makes it slightly easier or slightly harder than a clean item. The threshold to drop is two or more flaws on the same item, because that is where the failure modes start interacting (a non-homogeneous option list with absolutes in two options, for example, is essentially a length-tell wrapped in a logic puzzle). On a Step 1 deck of two hundred AI-generated items, expect five to fifteen to have one flaw and one or two to have two or more; remove the two-plus items, keep the one-flaw items in the rotation, and note the flaw so you do not pattern-match on it during revisit.

studyly.io/quality. It shows the four named tools on the held-out three-document eval (Studyly 81.3, Unattle 78.0, Gauntlet 68.0, Turbolearn 57.8) on the four-criterion 0/3/7/10 rubric. The numbers correlate with what you would get if you ran this thirteen-point rubric on ten consecutive items from each tool, but the leaderboard is the composite score, not the per-flaw breakdown. For the underlying four-criterion methodology see studyly.io/t/ai-study-question-quality-rubric.