14 · Sets: uniqueness, membership, set algebra
Deduplicate in one call, check membership in O(1), and find label mismatches with a single subtraction.
A set is an **unordered collection of unique elements** — adding the same value twice has no effect, and `in` checks run in O(1) regardless of size.
Without this:
Without sets, deduplicating a list of training IDs requires a loop with a 'seen' tracker; checking whether a label exists in a vocabulary requires scanning the whole list — O(n) per lookup.
A set literal uses curly braces: {1, 2, 3}. Important: {} creates an empty dict, not a set — use set() for an empty set. Sets are mutable (.add(), .remove(), .discard()), unordered (no indexing), and only store hashable values. The key power is O(1) membership testing and the built-in set algebra operators.
Python (in browser)
The round-trip `list(set(x))` is idiomatic for deduplication, but remember the output order is not guaranteed. Sort afterward if order matters.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`-` is set difference, `&` is intersection, `|` is union, `^` is symmetric difference. These read exactly like Venn diagram operations.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
At 50 000 elements the set is typically 100–500× faster. The gap widens linearly with list size — at 500 000 elements the list is ~10× slower still.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
What does `{1, 2, 2, 3} == {1, 2, 3}` return?
- `set()` creates an empty set — `{}` is an empty dict. Sets store unique hashable values and support O(1) membership tests.
- Set algebra: `|` union, `&` intersection, `-` difference, `^` symmetric difference. These are the Venn diagram operations.
- Sets are unordered — never depend on iteration order. Use `dict.fromkeys()` for unique values with stable order.
Deduplicating a list of training file paths before loading: `unique_paths = list(set(all_paths))`. Comparing label vocabularies between train and test splits: `set(train_labels) - set(test_labels)` reveals classes the model will never be evaluated on. Building a word vocabulary: `vocab = set(token for sentence in corpus for token in sentence.split())`.
If you remove it: Without sets, every membership check on a vocabulary of 50 000 tokens is O(n) — for a corpus of 1 million sentences that would add roughly 50 billion comparisons to tokenization alone.