Back to the tutorial

The learning study

If your result differs.

Start by comparing the research question, data and analysis choices. The reference uses a two-sided Welch comparison of group means. The tutorial leaves that choice open for discussion, so a different method need not be an error. Exact numerical agreement is expected only for matching inputs and settings.

What you see What to check
Different sample size There must be 80 unique IDs, exactly 40 per group, no missing values and no exclusions. Check the original CSV and saved row selection.
A negative difference or d Check group order. Reversing the groups reverses the difference, t and d, and negates/reorders CI limits; the two-sided p-value stays the same.
Different df, p or CI Check which method and settings your agent used, and why. The reference uses independent Welch, two-sided inference and 95% confidence. Different people are in the two groups.
A 67.47 versus 67.48 mean The unrounded mean is 67.475. Decimal-tie and floating-point display rules can differ. Compare full precision and calculate from it.
Reported mean difference does not equal subtraction of displayed means The difference is calculated before rounding. Exact means 67.475 − 61.825 give 5.65.
An interval next to d Ask which estimand it describes. Our [0.86, 10.44] CI belongs to the score-point mean difference; no d interval was requested.
id called a scale variable Storage-based type inference cannot establish a variable's scientific role. The study sheet identifies it as a participant ID.
Agent switches tests after Shapiro or drops an outlier Ask it to justify the scientific decision. The reference retains all cases and did not select its test from a preliminary-test result.
Plot looks different Check all 80 scores, category order, summary type and axis limits. A median boxplot, a mean/CI plot and a scatter of individual scores encode different things.
Replay refuses to run Keep the whole project including hidden .nlss/ files. Exact replay also needs compatible NLSS code, R/packages and the recorded environment; do not bypass a failed gate.

If packages are missing, let the agent explain the specific requirement and follow the NLSS installation/dependency process. Never treat a failed run or an agent's plausible-looking answer as computed evidence. Keep the original file; use a new project if you deliberately change inputs or the research question.

The published reference results belong to the reference package, not the minimal two-file starter package. Open them on the website or from the separate reference package. No exact-match agent transcript is expected. Material 1.0 · © 2026 Mike Hammes / NLSS, CC BY 4.0.