Analyzing a typical user study

This vignette walks through the workflow colleyRstats was built for: a within-subjects user study, from raw data to the text and figures that go into the manuscript. Every number in the paper should come out of this script – no retyping, no copy-paste drift.

The study data

Twenty-four participants experienced three interface conditions and rated their mental demand (0–100, NASA-TLX style) and trust (1–7 Likert) after each. In your project this data frame comes from your logging or survey export; here we simulate it.

set.seed(42)
n <- 24

main_df <- data.frame(
  Participant = factor(rep(seq_len(n), each = 3)),
  ConditionID = factor(rep(c("Baseline", "HUD", "LED"), times = n))
)

cond_effect <- c(Baseline = 55, HUD = 46, LED = 44)[as.character(main_df$ConditionID)]
main_df$tlx_mental <- pmin(100, pmax(0, round(cond_effect + rnorm(nrow(main_df), sd = 10))))
main_df$trust <- pmin(7, pmax(1, round(3.5 +
  (main_df$ConditionID != "Baseline") * 0.9 + rnorm(nrow(main_df), sd = 1))))

Two things matter before any analysis:

the participant and condition columns must be factors (they are above);
decide up front whether the design is within- or between-subjects – it changes the tests, the plots, and the effect sizes.

The quick route: one call per dependent variable

analyze_and_report() runs the whole per-DV pipeline: it checks the assumptions (and phrases the justification for the methods section), builds the matching ggstatsplot figure with automatic parametric/non-parametric selection, reports the omnibus test, and – for three or more groups – the significant post-hoc comparisons.

res <- analyze_and_report(
  main_df,
  dv = "tlx_mental", iv = "ConditionID",
  design = "within",
  ylab = "Mental Demand (TLX)"
)
#> Shapiro--Wilk tests indicated no significant deviation from normality in any group (all $p \geq 0.05$); therefore, parametric tests were used.
#> A ANOVA estimation for factorial designs using 'afex' found a significant effect of \ConditionID on tlx\_mental (\F{1.92190837641126}{44.2038926574591}{5.75}, \p{0.007}, r=0.11).
#> A Student's t post-hoc test found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to HUD (\m{45.88}, \sd{9.76}); \padj{0.033}).
#> A Student's t post-hoc test found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to LED (\m{45.21}, \sd{12.51}); \padj{0.015}).

The result carries everything separately, so you can place the pieces where they belong:

res$plot

res$methods  # for the Methods section
#> [1] "Shapiro--Wilk tests indicated no significant deviation from normality in any group (all $p \\geq 0.05$); therefore, parametric tests were used."
res$text     # the omnibus result
#> [1] "A ANOVA estimation for factorial designs using 'afex' found a significant effect of \\ConditionID on tlx\\_mental (\\F{1.92190837641126}{44.2038926574591}{5.75}, \\p{0.007}, r=0.11). "
res$posthoc  # significant pairwise comparisons (NULL for 2 groups)
#> [1] "A Student's t post-hoc test found that Baseline was significantly higher (\\m{54.92}, \\sd{11.10}) in terms of tlx\\_mental compared to HUD (\\m{45.88}, \\sd{9.76}); \\padj{0.033}). " 
#> [2] "A Student's t post-hoc test found that Baseline was significantly higher (\\m{54.92}, \\sd{11.10}) in terms of tlx\\_mental compared to LED (\\m{45.21}, \\sd{12.51}); \\padj{0.015}). "

For a whole questionnaire battery, report_all() does this for every scale and adds a Holm-corrected summary across the dependent variables:

battery <- report_all(
  main_df,
  dvs = c("tlx_mental", "trust"),
  iv = "ConditionID",
  design = "within",
  labels = c(tlx_mental = "Mental Demand", trust = "Trust")
)
#> Shapiro--Wilk tests indicated no significant deviation from normality in any group (all $p \geq 0.05$); therefore, parametric tests were used.
#> A ANOVA estimation for factorial designs using 'afex' found a significant effect of \ConditionID on tlx\_mental (\F{1.92190837641126}{44.2038926574591}{5.75}, \p{0.007}, r=0.11).
#> A Student's t post-hoc test found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to HUD (\m{45.88}, \sd{9.76}); \padj{0.033}).
#> A Student's t post-hoc test found that Baseline was significantly higher (\m{54.92}, \sd{11.10}) in terms of tlx\_mental compared to LED (\m{45.21}, \sd{12.51}); \padj{0.015}).
#> Shapiro--Wilk tests indicated a significant deviation from normality for at least one group (minimum $W = 0.79$, $p < 0.001$); therefore, non-parametric tests were used.
#> A Friedman rank sum test found a significant effect of \ConditionID on trust (\chisq(2)=10.93, \p{0.004}, r=0.23).
#> A Durbin-Conover post-hoc test found that HUD was significantly higher (\m{4.12}, \sd{1.08}) in terms of \trust compared to Baseline (\m{3.42}, \sd{1.10}); \padj{0.024}).
#> A Durbin-Conover post-hoc test found that LED was significantly higher (\m{4.46}, \sd{0.78}) in terms of \trust compared to Baseline (\m{3.42}, \sd{1.10}); \padj{0.003}).
battery$summary
#>           dv                                              method statistic
#> 1 tlx_mental ANOVA estimation for factorial designs using 'afex'  5.754671
#> 2      trust                              Friedman rank sum test 10.929577
#>       p.value      p.holm
#> 1 0.006577594 0.008466471
#> 2 0.004233235 0.008466471

Step by step, if you prefer control

1. Check the assumptions

check_normality_by_group(main_df, "ConditionID", "tlx_mental")
#> [1] TRUE
#> attr(,"tests")
#>   ConditionID         W   p_value
#> 1    Baseline 0.9709072 0.6894229
#> 2         HUD 0.9310723 0.1030198
#> 3         LED 0.9589363 0.4174263

assumption_methods_text() turns the same checks into the sentence reviewers expect next to the choice of test:

assumption_methods_text(main_df, x = "ConditionID", y = "tlx_mental")
#> Shapiro--Wilk tests indicated no significant deviation from normality in any group (all $p \geq 0.05$); therefore, parametric tests were used.

2. Plot with automatic test selection

The gg*WithPriorNormalityCheck* wrappers run the normality check and pick the parametric or non-parametric variant for you. The Asterisk versions annotate significant pairwise comparisons with APA-style stars instead of p-values:

plot_within_stats_asterisk(
  data = main_df,
  x = "ConditionID", y = "tlx_mental",
  ylab = "Mental Demand (TLX)",
  xlabels = c("Baseline", "HUD", "LED")
)
#> Scale for x is already present.
#> Adding another scale for x, which will replace the existing scale.

(Every function also has a snake_case alias – plot_within_stats_asterisk() is ggwithinstatsWithPriorNormalityCheckAsterisk().)

3. Factorial designs: the Aligned Rank Transform

When there is more than one factor and normality is violated, the standard non-parametric route is the Aligned Rank Transform (ARTool). reportART() turns the ANOVA table into LaTeX sentences, and reportArtCon() reports the pairwise contrasts with rank-biserial effect sizes:

m <- ARTool::art(
  tlx_mental ~ ConditionID + Error(Participant / ConditionID),
  data = main_df
)
reportART(anova(m), dv = "mental demand")
#> The ART found a significant main effect of \ConditionID on mental demand (\F{2}{46}{5.56}, \p{0.007}, $\eta_{p}^{2}$ = 0.19, 95\% CI: [0.04, 1.00]).

ac <- ARTool::art.con(m, ~ ConditionID, adjust = "holm")
reportArtCon(
  ac,
  data = main_df, iv = "ConditionID", dv = "tlx_mental",
  paired = TRUE, id = "Participant"
)
#> A post-hoc test found that tlx\_mental for the \ConditionID Baseline was significantly higher (\m{54.92}, \sd{11.10}) than for HUD (\m{45.88}, \sd{9.76}; \padj{0.016}, \rankbiserial{0.54}) and LED (\m{45.21}, \sd{12.51}; \padj{0.016}, \rankbiserial{0.64}).

4. Descriptives

reportMeanAndSD(main_df, iv = "ConditionID", dv = "tlx_mental")
#> %Baseline: \m{54.92}, \sd{11.10}
#> %HUD: \m{45.88}, \sd{9.76}
#> %LED: \m{45.21}, \sd{12.51}

Into the manuscript

Every reporter accepts sink_to = "results/tlx.tex" to write its sentences to a file your manuscript can \input{} – re-run the analysis and the paper updates itself. Figures go through save_paper_figure(), which uses publication presets (ACM-style single-column 3.33 in, full-width 7 in):

fig_path <- file.path(tempdir(), "tlx-mental.pdf")
save_paper_figure(res$plot, fig_path, columns = 2)
#> Saved figure to 'C:\Users\Mark\AppData\Local\Temp\RtmpOyntc5/tlx-mental.pdf' (7 x 4.66666666666667 in).

The LaTeX macros used by the reporters (\F, \p, \m, …) are defined by latex_preamble() or the shipped colleyRstats.sty; see vignette("overleaf") for the full R-to-Overleaf pipeline, including emit_overleaf(), which bundles the entire analysis into a folder that compiles as-is.