Metacognition and AI-generated practice questions

By Better Wise

Measuring the Gap Between Feeling and Knowing

Last lesson you built elaboration and dual coding links around new ideas. Those links are only worth something if they survive retrieval, so this lesson asks a harder question: does your confidence match what you can actually produce from memory?

By the end you should be able to:

• Predict your score before a test, then compare prediction with reality.
• Use self-explanation to locate the exact place where understanding breaks.
• Write AI prompts that produce practice questions instead of summaries.
• Start a calibration log you will fold into your 4-week plan.

Why Your Confidence Is a Poor Scorekeeper

Metacognition has two parts. Monitoring asks "do I know this?" and control decides what you do next: restudy, test again, or move on. Monitoring is the part that fools people.

Judgments of learning made immediately after study are inflated, because the material is still fresh and reading feels smooth. The same judgments made after a delay are noticeably more accurate (Nelson and Dunlosky, 1991). This is the fluency illusion wearing a lab coat: your confidence is a feeling, your score is a measurement.

Calibration is the match between predicted and actual performance. Most learners are overconfident on familiar material and underconfident on genuinely hard material. Overconfidence is not a character flaw, it usually means your test was too easy or too soon.

The loop below is the whole skill. Run it often enough and the prediction step gets realistic.

flowchart TD
  P["🔮<br/>Predict"] --> T["✍️<br/>Retrieve closed book"]
  T --> S["📊<br/>Score"]
  S --> C["🔍<br/>Compare and explain gap"]
  C --> A["🎯<br/>Adjust next review"]

Tip: Predict at the level of a specific question, not a whole chapter. "I will get 3 of these 5 right" produces a usable number. "I know this chapter" produces a mood.

Practice 1. Pick one topic from your project list. Before opening your notes, write a 0 to 100 percent confidence prediction for three specific questions, then answer them closed book.

Click to reveal guidance

The interesting number is the gap, not the score. A prediction of 80 with an actual 40 is a useful finding: you have a fluency pocket. Note the topic and the gap size, then schedule a shorter interval before the next review.

Self-Explanation as a Gap Detector

Self-explanation means saying, in your own words, why a step works and how the parts connect, while you study. A meta-analysis by Bisra and colleagues (2018) found a moderate, reliable benefit. The mechanism is simple: gaps show up as mumbling.

Useful stems:

• "Why does this step follow from the previous one?"
• "What would change if this condition were different?"
• "How would I explain this to someone who missed last week?"

Two rules make it work. First, explain before you check the answer, because explaining with the answer visible feels fluent for the wrong reason. Second, push for causal language: if you cannot say "because" or "therefore," you are probably describing rather than explaining.

Those explanations are also raw material. Every place you stumble becomes a candidate practice question later in this lesson.

Practice 2. Explain one worked example from your project out loud for 60 seconds, without notes. Mark every spot where you say "and then it just sort of..." Those are your gaps.

Click to reveal guidance

Aim for two or three flagged moments. Write each one as a question ("Why does the second step need the precondition from step one?"). You now have items for your AI prompt and a target list for your next session.

Prompting AI for Practice Without Outsourcing Recall

The division of labor is fixed: AI writes items, you retrieve answers. Two failure modes to avoid. Asking for a summary and reading it is passive review with extra steps. Accepting items without checking them means practicing on errors.

A prompt that produces usable practice contains five parts:

1. The source: paste the exact passage or name the scope precisely.
2. The item types: short answer, cloze, application, or "find the error."
3. One question at a time, with the answer withheld until you ask for it.
4. An answer key with a one-line rationale and a quote from your source.
5. A difficulty band near 85 percent correct.

Then verify each item before you use it. Is it true? Is it answerable from your source? Is there exactly one defensible answer? Language models produce fluent, confident, and sometimes wrong items, so the source is the authority, not the model.

You can also ask the AI to compare your written answer against the source and list what you omitted. You still judge the comparison, because you are the one building the memory.

Practice 3. Write a three-sentence prompt for one topic in your plan, run it, and reject or repair at least one item.

Click to reveal a model prompt

"Here is my source text on [topic]: [paste]. Write one short-answer question at a time that requires applying the concept, not recalling a definition. Do not show the answer until I reply 'show'. When you do, give a one-sentence rationale and quote the sentence in my source that supports it."

Reject items that test trivia, contain two questions in one, or cannot be answered from the pasted text.

Keeping a Calibration Log for Your Project

You need one table and two minutes per session. Over four weeks it will tell you more about your study habits than any feeling will.

Read the pattern weekly. A gap above roughly 20 points across several sessions means your self-tests are too easy or too soon. A consistent negative gap means you are underestimating yourself and may be over-reviewing material you already hold.

Rules of Thumb That Save You From False Confidence

• Predict before you peek. A prediction made after seeing the answer is not a prediction.
• Test after a delay of at least a day when you want a trustworthy judgment of learning.
• If a question feels easy, check whether it is easy or merely familiar.
• Keep practice near 85 percent correct. Below 70 percent, add support. Above 95 percent, raise the difficulty.
• Treat AI output as a draft item bank, never as the answer key of record.
• One identified gap is a successful session. Chasing a perfect score usually means testing what you already know.

What to Carry Forward

• Confidence tracks fluency, not storage. Compare predictions with closed-book scores.
• Delayed judgments of learning are more accurate than immediate ones.
• Self-explanation before checking the answer locates gaps that rereading hides.
• AI is a question generator. The retrieval, and the verification, stay with you.
• A calibration log turns all of this into numbers you can act on.

Next, you will fold these habits into a 4-week learning and retention plan: spaced intervals, interleaved practice, an AI item bank you have verified, and a weekly calibration check.