NotebookLM flashcard quality MCQ eval: which feature you actually score, and what the rubric says.

No. NotebookLM ships flashcards and quizzes as two separate generators from the same uploaded source. Flashcards are 2-sided cards (term on the front, definition on the back) with an Explain button on the back. Quizzes are 4-option multiple-choice questions with an Explain button on the wrong answer. If you are running an MCQ eval against NotebookLM, you are scoring the quiz feature, not the flashcards. The Google announcement and the in-product UI both treat them as distinct surfaces.

On a 4-axis rubric (factual correctness, clarity, distractor quality, question-type coverage), NotebookLM's quiz output scores well on the first two and hits a ceiling on the second two. Factual correctness is strong because every question is grounded only in the sources you upload and the wrong-answer explanation cites the line it came from. Clarity is strong because language models are genuinely good at fluent stems. Distractor quality is mixed: the wrong options are on-topic but one is often eliminable on length or phrasing alone. Question-type coverage is the structural ceiling: the feature produces multiple-choice questions and nothing else, so it tests recognition but not application or case-based reasoning.

The studyly.io/quality leaderboard compared dedicated study-question generators on a held-out three-document eval: Studyly 81.3, Unattle 78.0, Gauntlet 68.0, Turbolearn 57.8. NotebookLM is a general-purpose research and document-Q&A tool that added quizzes and flashcards as one feature among many in September 2025; it is in a different product category. That does not mean its MCQ output cannot be scored on the same 4-axis rubric, only that the direct head-to-head number would carry a category caveat.

Upload one document, generate one quiz at standard size on hard difficulty, export the questions (screenshot or copy the stem and the four options into a sheet), then have a graded answer key for each. Score each question on four binary axes. (1) Factual: does the keyed answer match the source? (2) Clarity: is the stem unambiguous? (3) Distractors: are all three wrong options plausible and length-matched, or can one be eliminated without knowing the fact? (4) Type: is this a recognition question, an application question, or a case-style vignette? Tally the percentages. The factual and clarity scores will be high; the distractor and type scores will reveal the gap.

It changes the user experience but not the per-question quality scores. The April update added progress saving, got-it and missed-it marking, replay-only-missed, and card shuffling. Those are retention and review affordances, not generation-time changes. The MCQ that comes out of the generator is the same MCQ. Shuffle reorders cards so position-memorization breaks down, but it does not reword the stem on retake, which is the separate failure mode that turns a quiz into a memory test for the question instead of the material.

On the studyly.io/quality leaderboard, distractor quality is the single criterion with the largest tool-to-tool spread, because that is where in-flight gating beats post-hoc generation. Studyly's generator runs four checks per card before emission (source-anchoring, length-matching within 25 percent, filler-template ban, grammar parallelism), and the 23.5-point spread between the top and bottom of the leaderboard collapses substantially when you isolate the factual correctness axis. NotebookLM does not publish whether its quiz generator applies any of these gates. Inspection of its output suggests it does not enforce option-length matching, which is the distractor failure mode most students notice.

Depends on what your exam tests. For a vocabulary or definitions-style test (anatomy structures, drug names, dates), recognition practice is what you need and NotebookLM's MCQ output is enough. For a clinical, case-based, or application-heavy exam (USMLE Step 1 and 2 CK, NBME shelf exams, NCLEX next-gen items, bar exam essay-style fact patterns), recognition practice trains the easier half of the skill. The eval gap is structural, not a quality bug.

The flashcards score well on factual correctness for the same source-grounding reason, and on clarity because they are short by construction. The distractor-quality axis does not apply (there are no distractors on a 2-sided card). The question-type axis is collapsed: every flashcard is a single cognitive task, recall a definition from a term. If you are looking at NotebookLM's flashcards as a category, you are looking at one good cell in the type-coverage matrix, not at MCQs.

No. As of May 2026, Google publishes a Help page and a product-announcement blog post describing how the quizzes and flashcards are generated, but it does not publish any numeric quality benchmark for the quiz output: no factual-accuracy percentage, no distractor-quality score, no head-to-head against other tools. The only public numbered study-question eval is the four-axis leaderboard at studyly.io/quality, and NotebookLM was excluded from it because the bake-off compared dedicated question generators, not general-purpose research tools. So the honest answer to 'what does NotebookLM score on a question-quality eval' is qualitative (two axes clean, two axes capped), not a single published number. To get a number for your own material, run the four-axis rubric on ten consecutive NotebookLM quiz questions as described above.