Sentence tokenizer

Source: R/RcppExports.R

Get sentence tokens from text

tokenize_sentences(input, EOS = ".?!:;")

Arguments

input

a character vector.
EOS

a length one character vector listing all (single character) end-of-sentence tokens.

Value

a character vector, each entry of which corresponds to a single sentence.

Author

Valerio Gherardi

Examples

tokenize_sentences("Hi there! I'm using `sbo`.")

#> [1] "Hi there"        "I'm using `sbo`"