---
title: "2. Preprocessing"
output:
  rmarkdown::html_vignette:
    toc: true
bibliography: references.bib
link-citations: true
vignette: >
  %\VignetteIndexEntry{2. Preprocessing}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(
  collapse = TRUE, comment = "#>", message = FALSE, warning = FALSE,
  fig.width = 7, fig.height = 5.5
)
library(idiographic)
data(srl)
vars <- c("efficacy", "value", "planning", "monitoring", "effort")
```

The autoregressive models estimated in this package — ordinary and graphical
VAR, multilevel VAR, unified SEM, GIMME, and their rolling variants — share a
common set of assumptions about the input series. Each presumes weak
stationarity, meaning that the mean, variance, and autocovariance of every
series are constant across the observation window; linear, first-order (lag-one)
dynamics; measurement occasions that are correctly ordered and approximately
equally spaced; and non-degenerate variance in each variable
[@bringmann2013; @epskamp2018mlvar]. Intensive longitudinal data seldom satisfy
these conditions automatically. Series drift or trend across a protocol, their
variance differs between its early and late phases, occasions are missed or
unevenly attended, and an item is occasionally answered identically throughout.
None of these violations is detected by the estimators themselves: a trending
series, for instance, is fitted without error and its trend is absorbed into an
inflated autoregressive coefficient, so the resulting network misrepresents the
dynamics rather than failing outright.

The purpose of `preprocess()` is to make these assumptions inspectable, and
where necessary correctable, before a model is estimated and interpreted. It
constructs the same lag-one design that the estimators use — applying the same
variable selection, ordering, scaling, and within-person centring — and
evaluates each subject-series against the stationarity assumptions, returning a
set of diagnostic flags rather than a fitted network. It is not a required step:
the estimators accept the raw data and standardize and centre it internally. Its
role is diagnostic, and, where an assumption is violated, corrective, through
the `detrend` argument described below.

# The data

The estimators expect long format: one row per person-occasion, an id column, an
ordering column, and numeric time-varying indicators. The bundled `srl` data
hold self-regulated-learning indicators for 36 students measured over 156
occasions each; `name` is the student and `day` orders the occasions. This
vignette works with five indicators: `efficacy`, `value`, `planning`,
`monitoring`, and `effort`.

A temporal edge is only defined once the series is ordered: `from -> to` means
`from` at occasion $t-1$ predicts `to` at occasion $t$. Losing or reordering
lagged pairs changes that estimand, which is why the design is worth inspecting
before fitting.

# Running preprocess and reading the diagnostics

A call to `preprocess()` takes the variables and the identifier column. Printing
the returned object reports the retained design and, where an assumption is at
risk, an explicit recommendation for addressing it.

```{r run}
pp <- preprocess(srl, vars = vars, id = "name")
pp
```

The five indicators over 36 students give 5616 ordered rows. Of these, 5548
survive as complete current/lagged pairs: each student loses the first occasion
to the initial lag, and a few students lose a little more to missing values. No
series is constant, none shows near-unit-root persistence, and the unit-root
screen is clear — but ten subject-series trip the linear-trend flag, and the
message names the fix.

The `summary()` method aggregates the per-subject diagnostics to one row per
variable, reporting its mean dispersion and the number of subject-series that
trip each flag.

```{r summary}
summary(pp)
```

The trend flags are concentrated in `planning` (5 of 36 students) and `value`
(2), with one each elsewhere. Across the flagged students `planning` drifts by
roughly 0.003 standardized units per occasion — small per step, but enough over
156 occasions to bias its lag-one coefficients. The `n_high_ar` and
`n_unit_root` columns are zero throughout: the non-stationarity here is a gentle
deterministic trend, not a random walk.

The `counts` table gives the lag-pair accounting per student, which is the
denominator behind every temporal edge.

```{r counts}
head(pp$counts)
```

Each 156-occasion series offers 155 possible lag pairs; students with complete
data retain all 155, while a student with a few missing occasions (here, Bob)
retains fewer.

# What the flags mean

Each row of `pp$diagnostics` is one subject-series, and each flag marks a
different way stationarity can fail:

- **`flag_trend`** — a significant linear slope over time; the mean is not
  constant.
- **`flag_high_ar`** — a lag-one autocorrelation at or above 0.95; near-unit-root
  persistence.
- **`flag_unit_root`** — an ADF-style screen suggests the series may not revert
  to a stable mean.
- **`flag_mean_shift`** / **`flag_sd_shift`** — the first and second halves of the
  series differ in mean or in spread; drift or changing volatility.
- **`flag_zero_variance`** — a constant series, which no network can use.

`flag_stationarity_risk` is `TRUE` when any of these fire, and is the column the
`summary()` roll-up counts as `n_flagged`.

# Cleaning non-stationary series with `detrend`

`preprocess()` standardizes and centres, but it does **not** remove trends on its
own — detrending changes what a temporal edge means, so it is a modelling
decision the analyst should make deliberately. The `detrend`
argument makes that decision explicit:

- `"none"` (default) — diagnose only.
- `"auto"` — transform only the flagged series: difference a stochastic trend
  (unit root / near-unit-root), linearly detrend a deterministic trend, and
  leave every stationary series untouched.
- `"linear"` / `"difference"` — force one transform on every series.

`"auto"` is the option the diagnostic message recommended. It runs over all
subject-series and cleans only those that need it — no subsetting, one call.

```{r auto}
clean <- preprocess(srl, vars = vars, id = "name", detrend = "auto")
summary(clean)
```

The ten trend-flagged series are linearly detrended and the other 170 are left
as they were, so every `n_trend` count drops to zero. The retained-pair count is
unchanged because no series here needed differencing (which would cost the first
occasion of each block). One `planning` flag survives: it is a variance shift,
not a trend, and `"auto"` deliberately does not touch heteroscedasticity — that
is a substantive issue to weigh, not something to difference away.

When `"auto"`'s per-series choices are not what you want, `detrend` also takes a
**named vector** to set the method one variable at a time; unlisted variables
are left untouched. And `checks` restricts the screening to the risks you care
about — for example, flag only trends and unit roots and ignore variance drift.

```{r fine-control}
clean2 <- preprocess(srl, vars = vars, id = "name",
                     detrend = c(planning = "difference", value = "linear"),
                     checks  = c("trend", "unit_root"))
summary(clean2)
```

The transformed lag-pair design is available in `clean$pairs` and its current
and lagged matrices in `matrices(clean)`. The model-fitting functions take long
data rather than a `preprocess_result`, so an applied analysis must make the
same documented transformation to its long input before fitting; the audit does
not silently alter `srl` or any later model call.

# References