tokenize
Stentor API Reference / stentor-utils/src / tokenize
Function: tokenize()
tokenize(
message?,options?):string[]
Defined in: packages/stentor-utils/src/string/tokenize.ts:577
Tokenizes a message into an array of words, optionally removing stop words.
Will lowercase the message, trim whitespace, and remove common punctuation. Normalizes fancy Mac OS/iOS quotes and apostrophes to standard ASCII equivalents.
Parameters​
message?​
string
The message to tokenize.
options?​
TokenizeOptions = {}
Options for tokenization.
Returns​
string[]
An array of tokens (words).