Back to the tutorial

The learning study

Take the discussion further.

The study sheet supplies the design and variable meanings. Short prompts ask research questions without specifying methods or answer checklists. Let the agent propose an approach, ask for clarification where needed, and explain its choices. The learner's task is to understand and question that proposal.

The reference is one worked analysis. A different method is not automatically a mistake; first check whether it addresses the same question and respects the design. Reserve the detailed questions below for discussion after the attempt.

Questions that go beyond "significant or not?"

  1. What is the estimand? Distinguish the score-point mean difference, the pooled-SD standardization and the logit coefficient in the generator. The latter is conditional on a latent person effect. Nonlinearity prevents simply translating 0.28 into 28 points or into d.
  2. What does the design buy us? Balanced random assignment concerns group comparability; independent simulation concerns dependence between people. Neither provides a real sampling frame or evidence about actual learners.
  3. Why does the reference use Welch? The target is a mean difference without requiring equal population variances. A Shapiro or Levene p-value is not a switchboard for the entire analysis. Here, even unremarkable diagnostics cannot certify the design assumptions.
  4. Can we call the effect useful? The CI spans 0.86–10.44 points. Introduce a hypothetical minimum useful difference of 5 points after the analysis and ask what remains uncertain. Clearly label this a sensitivity discussion, not a preregistered threshold or an additional completed hypothesis test.
  5. What does overlap mean? Both groups span 42–87. The boxplots summarize individuals; the test addresses means. Ask why overlap is compatible with p = .021, and why a mean difference does not show each person's causal effect.
  6. What does an audit trail establish? Trace one number from prose to result JSON to request and data. Reproducibility and file integrity can coexist with a poor estimand or an overstated conclusion.

What the simulation leaves out

The generator models scores conditional on a person's latent success probability, with 100 exchangeable Bernoulli trials. It does not model shared item difficulty, classroom clustering, measurement invariance, reliability estimation, adherence, repeated tests, dropout or treatment heterogeneity. The between-person latent term creates extra score variation relative to one homogeneous binomial population. A bounded discrete outcome is not exactly normal; the t analysis uses an approximate sampling argument for means.

These simplifications make the example readable. They should remain visible, especially when transferring a workflow to psychological research. A real study might need a different measurement model or dependence structure.

A lightweight learning check

Ask the learner to explain, without copying the reference:

  • what one row represents and why a paired test is inappropriate;
  • the difference's direction, units and uncertainty;
  • one incorrect p-value/CI interpretation;
  • why the result cannot establish real-world efficacy;
  • where the editable report and numerical evidence live.

Assess explanations and choices, not matching prose or reproducing a checklist of statistical terms. Let learners ask follow-up questions in their own words.

Material 1.0 · synthetic data · © 2026 Mike Hammes / NLSS, CC BY 4.0.