Two tools share the name "Step 1 question generator." Only one shows its work.

The word "generator" is hiding two different tools

Most guides on this topic treat "Step 1 question generator" as one category and then list options inside it. That framing is the mistake. The options do not differ by polish or price. They differ by something structural: whether the tool has a source document to build from.

An ungrounded prompt generator is a saved instruction you paste into ChatGPT or Claude. You give it a topic. It gives you questions. There is no file involved, so the model has to invent both the stem and the answer from whatever it absorbed in training. The output looks like a real NBME item. Whether it is correct is a separate question that the tool never answers.

A document-grounded generatorstarts from a file: a lecture deck, a First Aid chapter, a chapter PDF, a set of notes. It extracts the testable facts from that file and writes questions anchored to them. The answer is not the model's opinion; it is what the source says. When you miss a question, the tool can hand you back the exact slide or page the fact came from. That single property, a source you can return to, is the whole difference.

What the most prominent "Step 1 question generator" actually is

This is not a strawman. Below is the literal instruction behind the generator that comes up first when you look this up, a saved prompt published on docsbot.ai. It is a clean example of the ungrounded pattern, and reading it once tells you exactly what the category can and cannot do.

There is nothing wrong with the prompt as a prompt. It is well formed, it asks for four options and an explanation, it covers the right subjects. The problem is upstream of the wording: the only input is a topic name. With no source document, the model fills the stem and the key from training data, and no later step compares that key to anything. The generator cannot be more reliable than the model's memory of pathology, and it has no mechanism to flag the moments when that memory is wrong.

Why this matters more for Step 1 than for most exams

For a low-stakes quiz, an occasional wrong answer key is an annoyance. For Step 1 it is a specific kind of damage. The whole point of practice questions is repetition: you drill a fact until it is automatic. If the fact you are drilling is wrong, repetition does not protect you, it cements the error. You will recall it fast and confidently on exam day, and it will be wrong.

An ungrounded generator is most dangerous exactly where you need it most: the dense, detail-heavy corners of pharmacology, biochemistry, and microbiology where you are not sure of the answer yourself. If you already knew the material cold, you would catch the bad key. You use a generator because you do not, which means you are the least equipped person to audit its output. The tool that most needs a verification step is the one being used by the person least able to supply one manually.

So the real question to ask a Step 1 question generator is not "does it write good-looking vignettes." They all do. The question is "when this answer key is wrong, what catches it?" For an ungrounded prompt the honest answer is nothing. For a grounded generator the answer is the source document and the rubric, which is what the rest of this page is about.

The four checks a Step 1 question should pass before you drill it

A question is worth your time only if it survives four checks. These are the criteria Studyly scores every generated item on, and they double as a checklist for auditing any generator's output, including an ungrounded one.

The first item is the one an ungrounded generator cannot satisfy by construction. Items two through four are about craft and can, with effort, be hit by a good prompt. Item one is about provenance, and provenance is not something a prompt can add after the fact. Either the question was built from a source you can open, or it was not.

How to generate Step 1 questions you can trust

The workflow is short. Pick the source document closest to what you actually need to know. For school block exams, that is the professor's lecture deck for the block. For the boards-wide blueprint, it is a First Aid section or a chapter PDF. For a topic you keep missing in UWorld, it is your own typed notes on that topic.

Upload it. Studyly converts a roughly 90-slide deck into about 60 to 80 questions in around 60 seconds, in multiple-choice, free-response, and case-style formats, plus image-occlusion cards on any labelled figures. Drill in short five-minute passes rather than one long sitting. When you miss an item, open the cited slide or page instead of arguing with the explanation, because the explanation points at a document you can read.

On revisit, the same fact comes back rephrased: a different demographic, a different presenting detail, a reordered option list. That matters because the failure mode of any question you have seen before is recognizing the wording instead of recalling the fact. An ungrounded prompt hands you the identical stem every time, which lets you pattern-match your way to a streak that does not survive contact with the real exam. The deeper write-up on that rephrase loop is in the vignette drill guide.

When an ungrounded prompt is fine, and when nothing beats UWorld

The honest version of this argument has to admit two things. First, an ungrounded prompt generator is genuinely fine for warm-up volume and for testing whether you can recall a topic at all. If you ask ChatGPT for ten quick glycolysis questions and you miss six of them, you have learned something useful about your prep regardless of whether all ten keys were perfect. Use it as a thermometer, not as a textbook.

Second, a generator of either kind is not a replacement for UWorld or AMBOSS. Those banks have the strongest item editors in the field: calibrated distractors, physician-written explanations, and items reviewed against the published content outline. For first-pass and second-pass timed-block work they are the standard, and they should be. A grounded generator does not compete with them on item-writing polish.

Where a grounded generator wins is the territory the commercial banks structurally cannot cover: your school's specific lectures, the notes only you wrote, and the situation where you have already exhausted the finite commercial banks and need fresh items on your own material. Use UWorld and AMBOSS for the blueprint and the timed format. Use a document-grounded generator for everything your own documents cover and the banks do not. Skip the ungrounded prompt for anything you intend to actually memorize.

Related on this site: USMLE vignette drill from your lectures (the slide-to-clinical-stem transformation behind the questions), and the May 14, 2026 Step 1 format change (how the fourteen 30-minute blocks change how you should drill).