--- title: "Tuning and Pipelines" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Tuning and Pipelines} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>", fig.width = 7, fig.height = 5 ) ``` ```{r setup} library(tidylearn) library(dplyr) ``` ## Overview `tl_model()` fits one model with the hyperparameters you name. Two families build on that: - **Tuning** searches a space of hyperparameters and returns the model fitted with the winner. `tl_tune_grid()` walks every combination; `tl_tune_random()` samples from ranges. - **Pipelines** bundle preprocessing, a set of models, and an evaluation scheme into one object you can run, save, reload, and predict from. The two compose: tune to find the settings, then put the winning settings in a pipeline so the whole recipe is reproducible. ## Grid Search `tl_tune_grid()` takes a named list of candidate values, one entry per hyperparameter, and cross-validates every combination. ```{r} tuned_tree <- tl_tune_grid( iris, Species ~ ., method = "tree", param_grid = list(cp = c(0.001, 0.01, 0.1), minsplit = c(5, 20)), folds = 3, verbose = FALSE ) ``` What comes back is an ordinary tidylearn model, already fitted with the winning settings, so everything you would normally do with a model still works: ```{r} print(tuned_tree) ``` The search itself is attached as a `"tuning_results"` attribute: ```{r} tuning <- attr(tuned_tree, "tuning_results") names(tuning) ``` ```{r} tuning$results ``` ```{r} # The settings that won, and the score they won with tuning$best_params tuning$best_metric ``` The winning values reach the underlying fit, not just the report: ```{r} tuned_tree$fit$control$cp ``` ### Choosing the metric Without a `metric`, tuning uses accuracy for classification and RMSE for regression. Name one explicitly and the optimisation direction follows from the metric: `rmse`, `mse`, `mae` and `mape` are minimised, everything else maximised. Pass `maximize` only to override that. ```{r} tuned_reg <- tl_tune_grid( mtcars, mpg ~ ., method = "forest", param_grid = list(mtry = c(2, 4), ntree = c(100, 300)), folds = 3, metric = "rmse", verbose = FALSE ) attr(tuned_reg, "tuning_results")$best_params ``` ### Starting from a default grid `tl_default_param_grid()` supplies a reasonable starting grid per method, at three sizes. Use it as a first pass, then narrow around whatever won. ```{r} tl_default_param_grid("tree", size = "small") ``` ```{r} tl_default_param_grid("forest", size = "medium") ``` ```{r} tuned_default <- tl_tune_grid( iris, Species ~ ., method = "tree", param_grid = tl_default_param_grid("tree", size = "small"), folds = 3, verbose = FALSE ) attr(tuned_default, "tuning_results")$best_params ``` ## Random Search Grid search cost is the product of the candidate counts, so it grows quickly. `tl_tune_random()` samples `n_iter` points instead. How `param_space` describes each parameter decides how it is sampled: | Specification | Sampled as | |---|---| | Two numbers, `c(lo, hi)` | Uniform on the continuous interval | | Three or more numbers | Drawn from exactly those values | | A character vector | Drawn from those levels | | A function of no arguments | Whatever the function returns | A two-element vector is always continuous, so `minsplit = c(2, 40)` samples values like 11.34. For a parameter that has to be a whole number, list the candidates instead: ```{r} tuned_random <- tl_tune_random( iris, Species ~ ., method = "tree", param_space = list( cp = c(0.0001, 0.2), # continuous minsplit = c(2, 5, 10, 20, 30, 40) # drawn from these six ), n_iter = 8, folds = 3, seed = 42, verbose = FALSE ) attr(tuned_random, "tuning_results")$best_params ``` Pass `seed` whenever you want the search to be reproducible. Without it, two runs sample different points and can pick different winners. ## Looking at the Search `tl_plot_tuning_results()` reads the attribute and draws it four ways. ```{r} tl_plot_tuning_results(tuned_tree, plot_type = "scatter") ``` `"grid"` draws the two-parameter heat map that grid search is built for: ```{r} tl_plot_tuning_results(tuned_tree, plot_type = "grid") ``` `"parallel"` puts every parameter on its own axis, which scales past two: ```{r} tl_plot_tuning_results(tuned_tree, plot_type = "parallel") ``` `"importance"` ranks parameters by how much of the score variation each one explains — a quick read on which knob is worth refining: ```{r} tl_plot_tuning_results(tuned_tree, plot_type = "importance") ``` Every one of these is a ggplot2 object, so add to it as usual. ## Pipelines A pipeline records preprocessing, the models to fit, and how to evaluate them. Building it does no work; `tl_run_pipeline()` does. ```{r} split <- tl_split(iris, prop = 0.7, stratify = "Species", seed = 42) pipe <- tl_pipeline( split$train, Species ~ ., preprocessing = list(standardize = TRUE, dummy_encode = FALSE), models = list( tree = list(method = "tree"), forest = list(method = "forest", ntree = 300) ), evaluation = list( validation = "cv", cv_folds = 3, metrics = c("accuracy", "f1"), best_metric = "accuracy" ) ) print(pipe) ``` Anything you leave out of `preprocessing` or `evaluation` takes its default, so a partial list is fine. An unrecognised name is an error rather than a step that quietly does nothing. ```{r, error = TRUE} tl_pipeline(split$train, Species ~ ., preprocessing = list(scale_method = "standardize")) ``` ### Running it ```{r} run <- tl_run_pipeline(pipe, verbose = FALSE) names(run$models) ``` ```{r} print(run) ``` `tl_get_best_model()` returns the model that won on `best_metric`: ```{r} best <- tl_get_best_model(run) best$spec$method ``` ### Predicting through the pipeline This is the reason to use a pipeline rather than a bare model. Predicting on raw new data replays the preprocessing the pipeline learned during the run, applying the *training* centre and scale rather than recomputing them from the new rows. ```{r} preds <- tl_predict_pipeline(run, new_data = split$test, model_name = "forest") head(preds) ``` ```{r} mean(preds$.pred == split$test$Species) ``` Omit `model_name` to predict with the best model. ### Saving and reloading ```{r} path <- tempfile(fileext = ".rds") tl_save_pipeline(run, path) reloaded <- tl_load_pipeline(path) names(reloaded$models) ``` ```{r} # Predictions survive the round trip, preprocessing included reloaded_preds <- tl_predict_pipeline( reloaded, new_data = split$test, model_name = "forest" ) identical(reloaded_preds$.pred, preds$.pred) ``` ```{r, include = FALSE} unlink(path) ``` ## Tuning into a Pipeline Tuning tells you the settings; the pipeline holds them alongside the preprocessing that produced them. ```{r} tuned <- tl_tune_grid( split$train, Species ~ ., method = "forest", param_grid = list(mtry = c(2, 3), ntree = c(100, 300)), folds = 3, verbose = FALSE ) best_params <- attr(tuned, "tuning_results")$best_params best_params ``` ```{r} final <- tl_pipeline( split$train, Species ~ ., models = list( forest = c(list(method = "forest"), best_params) ), evaluation = list(cv_folds = 3, metrics = "accuracy", best_metric = "accuracy") ) final_run <- tl_run_pipeline(final, verbose = FALSE) final_preds <- tl_predict_pipeline(final_run, new_data = split$test) mean(final_preds$.pred == split$test$Species) ``` ## Cost Tuning multiplies fits. A grid of *g* combinations at *k* folds is *g × k* fits, plus one more to build the final model. The tree grid at the top of this vignette is 6 combinations × 3 folds = 18 fits, 19 with the final model, of a method that takes milliseconds. The same grid on `method = "xgboost"` with 1000 rounds is the same 19 fits of something much slower. Two levers, in the order worth pulling: 1. **Fewer folds.** Going from 5 to 3 removes 40% of the work and still gives out-of-sample estimates. 2. **Random over grid.** `tl_tune_random(n_iter = 10)` costs a fixed 10 points regardless of how many parameters you are searching, where a grid over the same parameters costs their product. `tl_compute_advisor()` estimates the cost of a single fit before you multiply it by the search — worth a look before starting a long grid. ## Where to Go Next - **AutoML** (`vignette("automl")`) searches across methods rather than within one, and manages its own budget. - **Diagnostics** (`vignette("diagnostics")`) covers what to check once you have a fitted model.