step_sentence_embeddings() converts a character column
into dense embeddings from a pretrained model. The step stores a
reference to the model rather than the model itself, checks that the
embedding dimension is unchanged when baking, and caches embeddings
within the R session. The embedding model is chosen with an engine, see
the new “Sentence embeddings” vignette. (#303)
embedding_custom() supplies your own embedding
function.embedding_ollama() gets embeddings from a running
Ollama server using httr2.embedding_huggingface() runs a sentence-transformers
model from the Hugging Face Hub in Python through reticulate. The model
is pinned with a required revision, and the loaded model is
kept for the R session but never stored in the recipe.embedding_model2vec() embeds text with a downloaded
Model2Vec model entirely in R, without Python or a server. The model is
read once per R session. (#301)step_token_replace() replaces tokens in a token
variable according to pattern and replacement
vectors, matching either exactly or with regular expressions.
(#134)step_tokenize(), step_tokenize_bpe(),
step_tokenize_sentencepiece(), and step_stem()
now validate their options argument with
recipes::check_options() and give informative errors for
unnamed, non-list, or disallowed elements. (#291)
step_tokenize() documentation now states that
training_options also applies to
engine = "udpipe", where it must contain the loaded
model, and includes an example. (#237)
step_tokenize_sentencepiece() gains a
model argument, which takes the path to a pretrained
sentencepiece model to use instead of training a new one.
(#278)
step_pos_filter() gains a remove_tags
argument, which removes tokens with the given part of speech tags
instead of keeping them. It can’t be used together with
keep_tags. (#218)
New read_embeddings_model2vec() and
read_embeddings_bpemb() read downloaded Model2Vec and BPEmb
files into the tibble that step_word_embeddings() takes as
embeddings. New
download_embeddings_model2vec() downloads a Model2Vec
model. (#303)
The counting functions in count_functions, used by
step_textfeature(), are now faster. The word-list counts
(first_person(), prepositions(), etc.) are
about 2x faster, the n_uq_*() counts are 2 to 5x faster,
and n_uq_charS() uses less memory on large inputs.
(#267)
Argument documentation has been revised across all steps to
follow the tidyverse style guide: defaults are now stated, fixed sets of
values are listed, code is formatted consistently, and the descriptions
of res, smooth_idf, norm, and
vocabulary are now accurate. (#263)
Documentation now links to the external functions that do the
underlying work in several steps, such as
janitor::make_clean_names() in
step_clean_levels() and step_clean_names().
(#198)
New vignettes:
step_sentence_embeddings() with its engines. (#303)read_embeddings_*() helpers, and includes a section on
training your own embeddings with wordsalad’s glove(),
word2vec(), and fasttext() for use with
step_word_embeddings(). (#102, #103, #104, #303)All steps now work with 0 and 1 row data in bake().
This fixes errors in step_texthash(),
step_dummy_hash(), step_lda(), and
step_pos_filter() with 0 rows, and zero-length tokenlists
no longer lose their lemma and pos attributes, which broke
step_lemma(). (#290)
Custom tokenizer functions passed to custom_token
are now called positionally, so functions whose first argument isn’t
named x no longer error with “unused argument” at bake
time. (#248)
first_person(), first_personp(),
second_person(), second_personp(),
third_person(), to_be(), and
prepositions() (used by step_textfeature())
now tokenize the document into words and check whether it contains the
word. Previously they tested whether the word matched the entire
untokenized document, which almost always returned 0. (#316)
The counting functions used by step_textfeature()
are now more consistent (#329):
n_uq_urls() counts distinct full urls instead of being
capped at 2 distinct matches of the literal substring “http” or
“https”.n_charS() and n_uq_charS() now exclude
urls, hashtags, and mentions from the character count, as
documented.n_extraspaces() no longer counts a single tab or
newline as extra spaces. It only counts runs of 2 or more consecutive
whitespace characters.NA when given
NA input.show_tokens() now validates the upper bound of the
n argument (nrow(rec$template)) and gives a
clear error, instead of a confusing, unrelated one. (#322)
step_clean_levels() now restores the trained column
names for legacy trained objects, so bake() no longer
silently skips cleaning. (#321)
step_clean_levels() now gives character columns a
cleaning dictionary at prep() time, so bake()
uses a fixed lookup instead of applying
janitor::make_clean_names() to the batch of values being
baked. (#320)
step_dummy_hash() now restores the original columns
when keep_original_cols = TRUE and
collapse = TRUE, and no longer leaks the intermediate
collapsed column into the output. (#304)
step_dummy_hash() and step_texthash()
no longer crash (segfault) when num_terms = 0. They now
error cleanly during prep(). (#314)
step_lda() now produces deterministic topic features
at bake time. bake() reuses the vocabulary learned during
prep() and projects each document independently, so topic
weights sum to about 1 and no longer depend on what else is in the same
bake() call. (#315)
step_ngram() now errors clearly when
num_tokens or min_num_tokens is less than 1,
and when min_num_tokens is greater than
num_tokens, instead of surfacing an unrelated low-level
error. (#324)
step_ngram() no longer mislabels non-UTF-8
(e.g. latin1) input as UTF-8, which could cause nchar(),
toupper(), and other string operations on the output to
fail with encoding errors. (#323)
step_sequence_onehot() now errors clearly when
sequence_length is less than 1. (#324)
step_stem() no longer drops the lemma
and pos attributes carried by a tokenlist, so it can be
used before step_lemma() or step_pos_filter().
(#327)
step_texthash() no longer errors with “values value
must not be equal to the default 0” when baking with
sparse = "yes" and signed = TRUE (the default)
and signed hash collisions cancel out to zero. (#289)
step_tfidf() with sublinear_tf = TRUE
no longer produces negative values, because the sublinear transform is
now applied before normalization instead of after. (#317)
step_tokenize(), step_tokenize_bpe(),
step_tokenize_sentencepiece(), and
step_tokenize_wordpiece() now error informatively if a
tokenizer returns a list whose length doesn’t match the number of rows
in the input, instead of silently recycling and duplicating rows.
(#318)
step_tokenize() with engine = "spacyr"
or engine = "tokenizers.bpe",
step_tokenize_bpe(), and
step_tokenize_sentencepiece() no longer error with “unused
argument” in bake() when options is non-empty.
(#319)
step_tf() and step_tfidf()
documentation now states that the denominator for
weight_scheme = "term frequency" and
norm = "l1" is the number of in-vocabulary tokens, not the
total number of words. (#326)
step_word_embeddings() documentation now describes
the actual default prefix and column-naming behavior.
(#331)
bake() methods for steps that transform columns in
place now use recipes_map_cols(), which avoids copying the
data frame once per column and is much faster with many columns. This
requires recipes >= 1.4.0. (#312)sparse.
When set to "yes", they will produce sparse vectors. (#277)
step_dummy_hash()step_texthash()step_tf()step_tfidf()Documentation for tidy methods for all steps has been improved to describe the return value more accurately. (#262)
Calling ?tidy.step_*() now sends you to the
documentation for step_*() where the outcome is documented.
(#261)
step_textfeatures() has been made faster and more
robust. (#265)
step_clean_levels() where it would produce
NAs for character columns. (#274)textfeatures has been removed from Suggests. (#255)
step_textfeatures() no longer returns a politeness
feature. (#254)
step_untokenize() and step_normalization()
now returns factors instead of strings. (#247)step_clean_names() now throw an informative error if
needed non-standard role columns are missing during bake().
(#235)
The keep_original_cols argument has been added to
step_tokenmerge. This change should mean that every step
that produces new columns has the keep_original_cols
argument. (#242)
Many internal changes to improve consistency and slight speed increases.
Fixed bug where step_dummy_hash() and
step_texthash() would add new columns before old columns.
(#235)
Fixed bug where vocabulary_size wasn’t tunable in
step_tokenize_bpe(). (#239)
Steps with tunable arguments now have those arguments listed in the documentation.
All steps that add new columns will now informatively error if name collision occurs.
step_tf() wasn’t tunable for
weight argument.Setting token = "tweets" in
step_tokenize() have been deprecated due to
tokenizers::tokenize_tweets() being deprecated.
(#209)
step_sequence_onehot(),
step_dummy_hash(), step_dummy_texthash() now
return integers. step_tf() returns integer when
weight_scheme is "binary" or
"raw count".
All steps now have required_pkgs() methods.
if (require(...)) code.Remove use of okc_text in vignette
Fix bug in printing of tokenlists
step_tfidf() now correctly saves the idf values and
applies them to the testing data set.
tidy.step_tfidf() now returns calculated IDF
weights.
step_dummy_hash() generates binary indicators
(possibly signed) from simple factor or character vectors.
step_tokenize() has gotten a couple of cousin
functions step_tokenize_bpe(),
step_tokenize_sentencepiece() and
step_tokenize_wordpiece() which wraps {tokenizers.bpe},
{sentencepiece} and {wordpiece} respectively (#147).
Added all_tokenized() and
all_tokenized_predictors() to more easily select tokenized
columns (#132).
Use show_tokens() to more easily debug a recipe
involving tokenization.
Reorganize documentation for all recipe step tidy
methods (#126).
Steps now have a dedicated subsection detailing what happens when
tidy() is applied. (#163)
All recipe steps now officially support empty selections to be more aligned with dplyr and other packages that use tidyselect (#141).
step_ngram() has been given a speed increase to put
it in line with other packages performance.
step_tokenize() will now try to error if vocabulary
size is too low when using engine = "tokenizers.bpe"
(#119).
Warning given by step_tokenfilter() when filtering
failed to apply now correctly refers to the right argument name
(#137).
step_tf() now returns 0 instead of NaN when there
aren’t any tokens present (#118).
step_tokenfilter() now has a new argument
filter_fun will takes a function which can be used to
filter tokens. (#164)
tidy.step_stem() now correctly shows if custom
stemmer was used.
Added keep_original_cols argument to
step_lda, step_texthash(),
step_tf(), step_tfidf(),
step_word_embeddings(), step_dummy_hash(),
step_sequence_onehot(), and
step_textfeatures() (#139).
prefix argument now creates names according
to the pattern prefix_variablename_name/number. (#124)step_tokenfilter() and
step_sequence_onehot() that sometimes caused crashes in R
4.1.0.step_lda() now takes a tokenlist instead of a character
variable. See readme for more detail.step_sequence_onehot() now takes tokenlists as
input.step_tokenize().step_tokenize().step_clean_names() and
step_clean_levels(). (#101)step_ngram() gained an argument
min_num_tokens to be able to return multiple n-grams
together. (#90)step_text_normalization() to perform unicode
normalization on character vectors. (#86)step_word_embeddings() got a argument
aggregation_default to specify value in cases where no
words matches embedding.step_tokenize() got an engine argument to
specify packages other then tokenizers to tokenize.spacyr have been added as an engine to
step_tokenize().step_lemma() has been added to extract lemma attribute
from tokenlists.step_pos_filter() has been added to allow filtering of
tokens bases on their pat of speech tags.step_ngram() has been added to generate ngrams from
tokenlists.step_stem() not correctly uses the options argument.
(Thanks to @grayskripko for finding bug, #64)step_word2vec() have been changed to
step_lda() to reflect what is actually happening.step_word_embeddings() has been added. Allows for use
of pre-trained word embeddings to convert token columns to vectors in a
high-dimensional “meaning” space. (@jonthegeek, #20)step_tfidf() calculations are slightly changed due to
flaw in original implementation
https://github.com/dselivanov/text2vec/issues/280.step_textfeatures() have been added, allows for
multiple numerical features to be pulled from text.step_sequence_onehot() have been added, allows for one
hot encoding of sequences of fixed width.step_word2vec() have been added, calculates word2vec
dimensions.step_tokenmerge() have been added, combines multiple
list columns into one list-columns.step_texthash() now correctly accepts
signed argument.step_tf() and
step_tfidf().First CRAN version