The rubric behind the 23.5-point spread.

Factual correctness, clarity, distractor quality, and question-type coverage. The first three are scored per card on a 0/3/7/10 anchor scale. The fourth is scored per deck (the mix of MCQ vs free-response vs case-style vs image-occlusion across the whole output, not per individual card). Adding the four gives a 0-40 raw score that is then renormalized to 0-100. Studyly publishes the leaderboard at studyly.io/quality, where 81.3 is the renormalized score on a held-out three-document eval.

Likert collapses the difference between a question that is mildly wrong and a question that teaches you the wrong fact, because the middle of the scale is overloaded. Anchors at 0, 3, 7, and 10 force the grader to pick a category: broken, weak, solid, exemplary. The gap from 3 to 7 is where most disagreement happens between graders, so the rubric prose for each criterion specifies what 3 and 7 look like as concretely as possible. In practice this halves inter-rater drift on the held-out eval compared to a five-point scale, which is consistent with what the published literature on rubric design (Snorkel AI, others) finds.

Because it is the one a student cannot self-grade. Clarity is visible (re-read the stem; is it ambiguous?). Distractor quality is visible (look at the four options; is one obviously wrong, are the lengths matched?). Question-type coverage is visible (count MCQ vs free-response across the deck). Factual correctness requires knowing the right answer, which is the exact thing the student is studying to learn. The rubric weights it the same as the other criteria, but the only way to verify it honestly is to require the generator to cite the source span the keyed answer was lifted from. No citation, the criterion fails at 0.

Yes, and it takes about 25 minutes per tool to do honestly. Pull ten consecutive cards from the output (not cherry-picked; consecutive). Open the source document. For each card, score the three per-card criteria on the 0/3/7/10 scale, citing the specific failure mode (the rubric body below tells you what each anchor looks like). Then score type-coverage once over all ten. Sum, divide by 40, multiply by 100. That number is comparable across tools as long as you run the same protocol. The 60-second-per-card pace is realistic if you already know the material; double that if you don't.

The NBME Item-Writing Guide at nbme.org/educators/item-writing-guide is the authoritative source for the underlying principles—distractors should be plausible and parallel, filler templates are item-writing flaws, grammar mismatches give the answer away. The guide is written for human item-writers preparing licensure exams. It does not specify a per-criterion 0-10 anchor scale because that is an implementation choice for an automated grader. This rubric ports the NBME principles into a four-criterion 0/3/7/10 scoring sheet that you can paste into a spreadsheet and run against any AI tool.

On the same three held-out source documents, the same four criteria, and the same per-card scoring sheet, Studyly scored 81.3 and Turbolearn scored 57.8. The 23.5-point spread is mostly on factual correctness and distractor quality; clarity and question-type coverage spread less. In a 200-question deck, the practical translation is: a tool scoring 57.8 hands you roughly forty cards that are wrong, ambiguous, or test-takeable on length tells; a tool scoring 81.3 hands you maybe twelve. You still spot-check, but the floor is higher.

A single card cannot have a 'type mix.' Type coverage is a property of the output as a whole: does the deck contain MCQ, free-response, case-style, and image-occlusion in something resembling a balanced ratio, or is it 200 single-best-answer MCQs and nothing else? Per-card grading would either make every card score the same (the tool's choice of mix is fixed) or make the criterion meaningless. So the rubric grades it once across the full sample, on the same 0/3/7/10 scale, and adds the result to the per-card totals.

Yes, on the per-card criteria, with one wrinkle. Factual correctness still anchors to a cited span (a slide region for image-occlusion, the source paragraph for free-response). Clarity translates: an image-occlusion card with three overlapping masks scores low on clarity in the same way an ambiguous stem does. Distractor quality only applies to multiple-choice formats; for free-response there is no distractor, so the criterion is replaced by 'rubric-of-the-rubric' (does a model answer exist that the student can grade against). Question-type coverage rewards the deck for containing all four formats.

studyly.io/quality. It shows the four named tools (Studyly 81.3, Unattle 78.0, Gauntlet 68.0, Turbolearn 57.8), the four criteria, and the held-out methodology. The number you see is the renormalized 0-100 score from the same rubric on this page. The leaderboard refreshes when new tools are added to the eval or when the held-out documents change.