AI-generated practice question quality: the part you cannot grade yourself.

Usually readable, not reliably correct. In a 2025 quality-assurance study published in BMC Medical Education, single-best-answer questions written by an AI were reviewed one by one: 22.2 percent were usable with no edits, 46.8 percent needed minor fixes, and 30.9 percent were rejected outright, some of them for plain factual errors. So a single AI practice question tends to look like a real exam item, because fluent question writing is the thing language models are genuinely good at. The catch is that 'looks like a real question' and 'has the right answer keyed' are two separate properties, and only the first one is visible to you.

Whether the keyed correct answer is actually correct. A practice question is graded on four things: clarity of the stem, plausibility of the distractors, the mix of question types across the set, and factual correctness of the answer key. The first three are visible. You can read a stem and tell if it is ambiguous. You can look at four options and see if one is a throwaway. You can count recall versus application across twenty questions. Factual correctness is different: to know the keyed answer is wrong, you have to already know the right answer, which is the exact thing you are studying to learn. You are the worst-positioned person to audit the answer key, because if you could, you would not need the practice question.

You cannot check it against your own memory, so check it against your source. For each question, find the exact slide, PDF page, or sentence the correct answer should trace to, then read that source and confirm the keyed answer matches what your professor actually said. If the tool cannot show you where the answer came from, that inability is itself the finding. The 60-second check on this page walks through it on ten questions: one wrong key in ten implies roughly twenty wrong keys in a 200-question set, and a wrong key does not just cost you that question, it teaches you the wrong fact.

Because most generators write the question from your material but answer it from their pretrained knowledge of the topic. The model has read thousands of textbooks. When it picks the correct option, it leans on that general knowledge rather than on the specific file you uploaded. If your professor's slide simplifies, updates, or contradicts the textbook consensus, the model keys the textbook answer, and the practice question now scores you wrong for repeating what your lecture actually taught. The failure is invisible at generation time and only surfaces on exam day.

Source-grounded means the keyed correct answer must trace to a specific span in the document you uploaded, not to the model's general knowledge. Studyly keeps a generated question only if its answer can be anchored to a slide, PDF page, or transcript timestamp in your upload; if no span can be cited, the question is not kept. This is the structural fix for the wrong-answer-key problem, because there is no textbook fact pulling the key away from your lecture. It also makes the one dimension you could not self-grade checkable: every question links back to the exact slide its answer came from, so verifying the key is a one-click jump instead of a memory test.

On a held-out three-document eval run by Jungle, the company behind Studyly, Studyly scored 81.3 across four criteria: factual correctness, clarity, distractor quality, and question-type coverage. Unattle scored 78.0, Gauntlet 68.0, Turbolearn 57.8, with a field average of 67.9. Held-out means the source documents were chosen after the system was frozen, so the number reflects generalization rather than memorization of a tuning set. The full methodology and per-criterion breakdown are at studyly.io/quality. Treat it as the company's own measurement on a consistent rubric, not an independent audit.

Only partly. You can tell a model 'use only the attached file' and it will mostly comply on the question text. The answer key is harder, because the model still scores the options using its internal knowledge and a prompt cannot verify that the keyed answer was actually lifted from a span in your upload. The instruction degrades to a self-report, and a model will sometimes claim a fact came from slide 12 when it did not. Prompt craft moves the average, but the only thing that removes the wrong-key failure mode is a generator that rejects a question when the answer cannot be anchored to your source.