canpumf 0.5.2

New features

• Experimental list_statcan_pumf_catalogue() crawls the live Statistics Canada “Public use microdata” listing and returns one row per discovered survey edition (catalogue_id, Title, edition, format, url, product_url) — a discovery counterpart to the curated list_canpumf_collection() that picks up newly released PUMFs automatically. Editions offered in several formats collapse to a single preferred row (CSV / flat text first), but genuinely distinct surveys or file-types that share a reference year are kept as separate rows (e.g. the GSS cycle and the Giving/Volunteering survey both released in 2007, or the census individual/family/household/hierarchical files for one year). Surveys distributed only by Electronic File Transfer report url = "(EFT)".
• The full crawl is expensive (hundreds of requests), so list_statcan_pumf_catalogue() caches its result for the duration of the R session and reuses it on subsequent calls with the same arguments. Pass refresh = TRUE to re-scrape the live catalogue and replace the cached result, e.g. to pick up a newly released survey mid-session.
• Census PUMF editions are decoded from their cenNN / nhsNN filename prefix (the 2011 cycle shipped as the National Household Survey) and ind / fam / hous / hier file type into canonical "YYYY (individuals)"-style strings matching list_canpumf_collection(). This is forward-compatible: the 2026 census PUMF will resolve automatically once released.
• list_statcan_pumf_catalogue() now returns a SeriesTitle column (the plain-language series name matching the acronym) alongside an edition-specific Title. For umbrella products whose catalogue title is only the series name (e.g. the consolidated General Social Survey, or a census year’s individuals/hierarchical pair) the Title is synthesised as "<series> — <edition>", where the structural edition descriptor disambiguates colliding reference years ("General Social Survey — Cycle 16 (2002)", "Census of Population — 2021 (individuals)"); per-edition products keep Statistics Canada’s own title.
• get_pumf() now resolves download URLs from the scraped catalogue first for the series the crawler covers (GSS, SHS, SFS, CPSS, CIS, CHS, ITS, CCAHS), so a newly released edition is downloadable without a package update. Series the crawler deliberately does not cover — LFS and Census (which keep their dedicated paths) and the Giving/Volunteering surveys (SGVP, which Statistics Canada ships under reused zip names the umbrella crawl cannot disambiguate) — continue to resolve through the curated list_canpumf_collection(). URL resolution never triggers a live crawl; it reads the cached catalogue.
• The package now ships a frozen snapshot of the full catalogue crawl (inst/extdata/pumf_catalogue.rds). It is the terminal, always-available fallback for both URL resolution and list_statcan_pumf_catalogue(): a freshly installed package with no user cache and no network still resolves every supported survey’s download URL, so a change to the Statistics Canada website cannot silently break get_pumf() between releases. The shipped snapshot is regenerated at each release.

Robustness

• get_pumf(), get_pumf_connection() and pumf_metadata() now fail gracefully when Statistics Canada is unreachable: a download failure no longer raises an error but instead emits an informative message and returns NULL. list_available_lfs_pumf_versions() likewise returns an empty result with a warning rather than erroring, matching the existing behaviour of list_canpumf_collection() and list_statcan_pumf_catalogue().
• close_pumf(NULL) is now a no-op, so it can be called unconditionally on a get_pumf() result that may be NULL.
• When options(canpumf.cache_path = ) is not set, the package now notes this once when attached and again on the first download, explaining that data is written to a temporary directory (and discarded at the end of the session) and how to configure a persistent cache. The underlying behaviour is unchanged — without a cache path, data is stored in tempdir() for the session.

Bug fixes

• Surveys whose StatCan ZIP archives carry accented path names stored in CP437/Latin-1 without the UTF-8 flag (e.g. the Survey of Household Spending 2017, whose data live under a Data - Données/ folder) now extract correctly on Linux and Windows. Previously utils::unzip() either errored with “invalid multibyte string” (Windows) or silently dropped the affected files under a non-UTF-8 locale (Linux), so the survey failed to import with “No parseable metadata files found”. Extraction now uses zip::unzip() as the primary, locale-agnostic extractor on every platform (with the macOS ditto/system-unzip chain retained as a fallback for newer ZIP compression variants), giving uniform cross-platform behaviour. zip is a new dependency.

canpumf 0.5.1

New features

• Multi-module survey support. Surveys that ship several linked files sharing a respondent key are now modelled as several joinable tables in one DuckDB file. get_pumf() returns the survey’s primary (respondent-level) module and emits a one-time message listing the available sibling modules; pumf_module(tbl, "<module>") opens a sibling on the same connection so the two are joinable, and announces the shared join key. Each module’s join key is recorded in the registry (module_key) so it never has to be guessed (it varies: RECID, PUMFID, MICRO_ID, CASEID, IDNUM). Converted surveys include GSS cycle 16 / “Aging and Social Support” 2002 (MAIN + CG4 + CG6 + CR), GSS Time Use 1998/2010/2015/2022 (Main + Episode), the Survey of Household Spending 2017 (Interview + Diary, each with its own bootstrap weights), and the Giving/Volunteering/Participating cycles 1997–2010 (MAIN + GS/VD/GIVE/VOLNTR).
• close_pumf() now also accepts a DuckDB connection returned by get_pumf_connection(), closing it directly, in addition to a lazy dplyr::tbl() returned by get_pumf().
• New parse_pdf_codebook() metadata parser for StatCan bilingual PDF frequency codebooks. This recovers variable and value labels for surveys whose only machine-readable companion is the data file — notably CPSS cycle 1, which (unlike CPSS 2–6) ships no variables.csv. CPSS 1 now imports with full bilingual labels (parity with the other cycles) when pdftools is installed. Like the existing PDF data-dictionary parser, it is a label fallback that only fires when no command file or codebook CSV is found, and requires pdftools (Suggests).

Documentation

• New “Working with multi-module PUMF surveys” vignette showing how to load the primary module, open sibling modules with pumf_module(), join them inside DuckDB, and use get_pumf_connection() / close_pumf() directly.
• New “Bootstrap weights” vignette documenting the resampling method, how the weights are stored, stratification, estimating uncertainty, and the incremental re-run behaviour (reuse, adding replicates, and regeneration when rows are added).

Bug fixes

• get_pumf("LFS") (and other calls) no longer trigger spurious RStudio “Error in dbSendQuery(…)” Connections-pane popups. Transient internal DuckDB connections (status checks, write phases, BSW edits) are no longer registered in the RStudio Connections pane; only the final connection returned to the user is registered.
• add_bootstrap_weights() on an in-memory data.frame/tibble that already has replicate columns now extends the existing set (generating only the additional replicates) instead of regenerating a full set and producing duplicate column names. This matches the DuckDB-backed behaviour.
• add_bootstrap_weights() now handles rows added to a survey table that already has bootstrap weights correctly. Previously it generated replicates for the new rows in isolation (resampling only among the new rows), which is statistically wrong. It now deletes and regenerates the affected weights: every row when unstratified, or only the strata that gained rows when strata_cols are in effect (complete strata keep their existing weights).
• GSS Time Use 1998 now imports cleanly regardless of locale. Under a C locale (as in R CMD check) list.files() selected the Main module’s SAS PROC FORMAT, which injected categorical codes onto continuous clock-time, duration, decimal-hour and birth-year variables; these are now declared force_numeric so their values are preserved. In addition, merge_metadata() no longer warns about label conflicts that arise solely from lossy supplement parsers (SAS labels, PDF dictionary/codebook) — authoritative-source conflicts still warn.

canpumf 0.5.0

Major changes

• Data is now imported into DuckDB (breaking change, but only requiring slight modification of code)
• Adaptable metadata parsing registry
• Multiple more robust strategies to parse metadata
• Better data download and import mechanics
• Extensive test suite to prevent regressions and catch if StatCan re-releases data with changed metadata