2 · Tokenization
Split raw text into words, sentences, and custom regex patterns — the indispensable first step of every NLP pipeline.
Tokenization splits raw text into discrete units (words, subwords, sentences) — every downstream step assumes this representation.
Without this:
Your raw string is one giant token; no model can extract meaningful patterns.
Tokenization is the process of breaking a stream of text into smaller units called tokens. A token is the smallest meaningful chunk of text your pipeline will work with — most often a word, but it can also be a subword, a character, or a sentence depending on the task.
Why does tokenization matter? Because virtually every NLP technique — TF-IDF, Naive Bayes, Word2Vec, BERT — operates on a sequence of tokens, not on a raw string. Tokenization is the interface between human language and machine-readable representations.
NLTK offers three tokenizers for different needs:
| Tokenizer | Best for |
|---|---|
| word_tokenize | General punctuation-aware word splitting |
| sent_tokenize | Splitting a paragraph into sentences |
| RegexpTokenizer | Custom rules (e.g. strip all punctuation) |
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Subword tokenization preview — deep dive in the Deep Learning track
What does `word_tokenize("don't")` produce in NLTK?
- `word_tokenize` is punctuation-aware and handles contractions; `sent_tokenize` splits into sentences; `RegexpTokenizer(r'\w+')` drops all punctuation.
- Never use `str.split()` for NLP — it creates noisy tokens like `"Hello,"` that pollute your vocabulary.
- Modern Transformers use subword tokenization (BPE, WordPiece, SentencePiece) to handle rare and unknown words — deep dive in the Deep Learning track.
Every Hugging Face model has a `Tokenizer` class; sklearn's `CountVectorizer` does its own internal tokenization. Understanding tokenization lets you customize both.
If you remove it: Without tokenization, your text is one opaque string — no frequency count, no TF-IDF weight, no embedding lookup is possible.