The learning study
If your result differs.
Start by comparing the research question, data and analysis choices. The reference uses a two-sided Welch comparison of group means. The tutorial leaves that choice open for discussion, so a different method need not be an error. Exact numerical agreement is expected only for matching inputs and settings.
| What you see | What to check |
|---|---|
| Different sample size | There must be 80 unique IDs, exactly 40 per group, no missing values and no exclusions. Check the original CSV and saved row selection. |
| A negative difference or d | Check group order. Reversing the groups reverses the difference, t and d, and negates/reorders CI limits; the two-sided p-value stays the same. |
| Different df, p or CI | Check which method and settings your agent used, and why. The reference uses independent Welch, two-sided inference and 95% confidence. Different people are in the two groups. |
| A 67.47 versus 67.48 mean | The unrounded mean is 67.475. Decimal-tie and floating-point display rules can differ. Compare full precision and calculate from it. |
| Reported mean difference does not equal subtraction of displayed means | The difference is calculated before rounding. Exact means 67.475 − 61.825 give 5.65. |
| An interval next to d | Ask which estimand it describes. Our [0.86, 10.44] CI belongs to the score-point mean difference; no d interval was requested. |
id called a scale variable |
Storage-based type inference cannot establish a variable's scientific role. The study sheet identifies it as a participant ID. |
| Agent switches tests after Shapiro or drops an outlier | Ask it to justify the scientific decision. The reference retains all cases and did not select its test from a preliminary-test result. |
| Plot looks different | Check all 80 scores, category order, summary type and axis limits. A median boxplot, a mean/CI plot and a scatter of individual scores encode different things. |
| Replay refuses to run | Keep the whole project including hidden .nlss/ files. Exact replay also needs compatible NLSS code, R/packages and the recorded environment; do not bypass a failed gate. |
If packages are missing, let the agent explain the specific requirement and follow the NLSS installation/dependency process. Never treat a failed run or an agent's plausible-looking answer as computed evidence. Keep the original file; use a new project if you deliberately change inputs or the research question.
The published reference results belong to the reference package, not the minimal two-file starter package. Open them on the website or from the separate reference package. No exact-match agent transcript is expected. Material 1.0 · © 2026 Mike Hammes / NLSS, CC BY 4.0.