5 · POS tagging & Named Entity Recognition
Classify each token's grammatical role and extract people, places, and organizations from raw text.
POS tagging classifies each token's part of speech; NER finds named entities (people, places, orgs) — both are structured outputs from raw text.
Without this:
You'd miss the structural information that powers every information-extraction system.
So far we've treated text as a bag of words — tokens without grammatical context. But language is structured: "bank" as a noun means a financial institution; "bank" as a verb means to tilt. "Apple" in an earnings report refers to the company; "apple" in a recipe refers to the fruit.
Part-of-Speech (POS) tagging assigns each token a grammatical tag: noun (NN), verb (VB), adjective (JJ), adverb (RB), etc. These tags are useful for:
- Providing the right POS hint to
WordNetLemmatizer - Keyword extraction (keep only nouns and verbs)
- Feature engineering for text classification
Named Entity Recognition (NER) is a step further: it identifies and classifies multi-token phrases that refer to real-world entities — people (PERSON), places (GPE), organizations (ORG), monetary values (MONEY), dates (DATE).
NLTK uses the Penn Treebank tagset for POS and a Maximum Entropy chunker for NER. These are solid for learning but slower and less accurate than modern alternatives like spaCy or transformer-based models.
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Penn Treebank tag reference — the 16 most useful tags
NER with nltk.ne_chunk — reference code (maxent_ne_chunker not available in Pyodide)
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
spaCy and Hugging Face — the production-grade NER alternatives to NLTK
What does the Penn Treebank POS tag 'NNP' mean?
- `nltk.pos_tag(tokens)` assigns Penn Treebank tags (NN, VB, JJ, RB, NNP, etc.) to each token — essential for correct lemmatization and noun extraction.
- NLTK's `ne_chunk` provides NER but requires the `maxent_ne_chunker` corpus which cannot be downloaded in Pyodide — use spaCy or Hugging Face for production NER.
- Filtering tokens by POS tag (keeping only `tag.startswith('NN')`) is a simple but powerful keyword extraction technique.
- The NNP tag is the bridge between tokenization and NER — proper nouns are the primary candidates for named entities.
Information extraction pipelines; chatbot intent detection (POS helps identify action verbs); resume parsers (NER extracts skills, companies, job titles); biomedical literature mining (drug names, gene symbols); financial news analysis (company names, monetary values).
If you remove it: Without POS tags, lemmatization defaults to noun — 'running' stays 'running' instead of reducing to 'run'. Without NER, extracting 'Apple', 'U.K.', '$1 billion' from text requires hand-written regex rules for every entity pattern.