Prompt-generated exam question quality is a property of the set, not the question.

Every guide grades the prompt. None grades the output.

Search how to make practice questions with AI and you get the same page forty times: a list of prompts, a note that detail matters ("generate algebra questions for high school" beats "math questions"), and a closing line telling you to review the output and treat the model as a co-creator. Every one of those guides assumes the same thing: that quality is a function of how well you phrase the request.

It is a reasonable assumption for the parts of a question a prompt can actually touch. A clearer prompt does produce a clearer stem. A prompt that bans filler options does drop "all of the above". A prompt that asks for matched-length distractors does get them most of the time. Three of the things that make a question good are wording instructions, and wording instructions transfer to a prompt cleanly.

The problem is the closing line. "Review the output" means read each question and decide if it is good. That review runs one question at a time, and the failure that decides whether a prompt-generated set is worth studying does not live inside any one question. It lives in the distribution of the set. You cannot read a distribution. You have to count it.

The recall skew, and why it is invisible per question

A prompt generates one question at a time. Pointed at a slide, the lowest-effort thing it can produce is a recall question: name the structure, define the term, pick the true statement. An application question needs an invented scenario, and a case-based question needs a coherent clinical or worked situation. That is harder, and it is exactly where studies found language models weakest. A multinational study in PLoS Onereported that ChatGPT "performed poorly when instructed to generate clinical scenario, possibly due to high complexity" (PMC10464959).

So a prompt asked plainly for "50 exam questions" drifts to the floor of Bloom's taxonomy. Each question still looks fine. The stem is clean, the distractors are plausible, nothing is obviously wrong. You read ten of them, they all check out, and you conclude the batch is good. It is not. You read ten recall questions and confirmed that recall questions are well-formed. You never measured the one thing that was broken.

Here is the shape researchers keep finding. The proportions below illustrate the direction, not measured percentages: what matters is that the two columns are different shapes.

The trap inside the trap: questions that fake their own level

There is a worse version. When you do ask a prompt for application questions, it will often give you a recall question with a scenario sentence bolted on the front. A patient, some lab values, a setting, and then a stem that still reduces to a single factual lookup. The question reads like application. It is graded, in your head, as application. It is recall.

This is a documented effect. Reviewers of AI-generated items have noted that a scenario can make an item "look like it has a cognitive level of application when it is really just a recall question". It is the single hardest failure to catch by reading, because the surface signal (a scenario is present) is the exact signal you use to tag the question as application.

The test that defeats it is subtraction. Delete the scenario sentence. A real application question collapses without its setup: no patient, no values, no answer. A fake one survives, because the scenario was decoration. If the stem still has one obvious answer after you strip the story, count it as recall, no matter how clinical it looked. For what a genuine case-style item should demand of you, see the vignette-drilling guide.

The 90-second count you run on your own batch

You do not need a benchmark to grade your own prompt-generated set. You need a tally. This takes about a minute and a half and tells you more than re-reading the questions ever will.

The read is binary. If your count roughly matches your exam's mix, the batch is usable and you should go drill it. If it is recall-heavy against an application-heavy exam, do not patch it question by question. Regenerate the whole set with an explicit type quota, recount, or move to a source-anchored generator that scores coverage before you ever see the output. Editing a recall-skewed set into a balanced one by hand costs more than regenerating it.

Why Studyly treats coverage as a score, not a feature

Studyly generates questions against the file you upload, your professor's actual slide deck or PDF, not a generic web question bank. That handles the factual-grounding column from the table above: a question is kept only if its answer traces to a span in your upload. The part that matters for this page is what happens at the level of the set.

The held-out eval scores four criteria, and question-type coverage is one of them, sitting next to factual correctness, clarity, and distractor quality. Treating coverage as a scored criterion is the structural reason a single Studyly upload comes back as a mix: multiple-choice, free-response, case-style, and image-occlusion questions from the same source, rather than 200 single-best-answer recall items. Coverage is graded at the level of the deck because that is the only level it exists at. For the per-question gates that run alongside it, see the in-flight rubric guide.

That is an internal measurement, not an independent audit, and it is worth reading it as one. But the framing is the point a raw prompt cannot replicate: quality is something you score on the whole set against named criteria, before the questions reach the student, not something the student reconstructs one item at a time at midnight.