40 · DataFrame: 2D labeled tables
A DataFrame is a dictionary of Series that share an index — the universal container for tabular ML data. Master creation, inspection, filtering, and the SettingWithCopyWarning footgun that trips every new Pandas user.
A **DataFrame** is a collection of Series that share the same row index — like a spreadsheet or an SQL table, but living in memory and directly interoperable with NumPy. The row index and column labels together mean every cell has a two-coordinate address: `df.loc[row_label, col_name]`. Understanding that a DataFrame is fundamentally a **dict of aligned Series** demystifies every DataFrame operation: selecting a column returns a Series; a boolean filter creates a mask Series; adding a column is just assigning a new key to the dict.
Without this:
Without DataFrames (or equivalent tooling), every preprocessing step — filtering rows, computing derived columns, selecting feature subsets — would require writing explicit loops over raw NumPy arrays and manually tracking column names via separate lists. The DataFrame gives you named columns, row-label alignment, and SQL-style operations (select, filter, join, group) in a single object. It is the primary reason Python displaced R as the default ML language.
A DataFrame is the 2D workhorse of pandas. Internally it is a dict of Series objects all sharing the same row index. You can think of it as a spreadsheet in RAM.
The most common ways to create a DataFrame:
# From a dict of lists — each key becomes a column name
pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]})
# From a list of dicts — each dict is one row
pd.DataFrame([{'name': 'alice', 'age': 30}, {'name': 'bob', 'age': 25}])
Key inspection methods to run on any new dataset:
.head(n)/.tail(n)— first/last n rows (default 5).shape—(n_rows, n_cols)tuple.info()— column names, dtypes, non-null counts, memory usage.describe()— summary statistics for numeric columns.dtypes— Series of dtype per column.columns— Index of column names
Column access:
df['col']— returns a Series (always safe)df[['col']]— returns a single-column DataFrame (double brackets)df.col— attribute-style access — avoid it for new code: it fails when column names contain spaces or clash with method names likedf.count.
Python (in browser)
`.describe()` is the first command to run on any new numeric dataset. It shows count, mean, std, min, quartiles, and max in one call — enough to catch outliers (max very far from mean), scale mismatches (age in years vs purchase in millions), and constant columns (std ≈ 0) before spending time on model training.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`df['age'] > 30` produces a boolean Series (the mask). Passing that mask back to `df[mask]` returns only the rows where the mask is `True`. Chaining `[['name', 'purchases']]` (double brackets) selects two columns as a DataFrame. This pattern — filter rows, then select columns — is the foundation of every data subset operation in a preprocessing pipeline.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Assigning to `df['new_col']` adds (or overwrites) a column in-place. The right-hand side can be a scalar, a list, a NumPy array, or a Series — pandas aligns by index automatically. This is the standard way to engineer new features before model training.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
`.loc[row_mask, col_name] = value` is the single safe idiom for conditional assignment. It resolves both the row selection and column assignment in one indexing operation, so pandas always modifies the original DataFrame. Chained indexing (`df[mask]['col'] = ...`) is the most common Pandas footgun — it silently fails or warns depending on whether the intermediate result is a view or a copy.
What is the difference between `df['col']` and `df[['col']]`?
- A DataFrame is a dict of aligned Series sharing the same row index. Each column is a Series; the shape is `(n_rows, n_cols)`.
- `.head()`, `.describe()`, `.info()`, `.dtypes` are the first four inspection commands on any new dataset.
- `df['col']` returns a Series; `df[['col']]` returns a single-column DataFrame. `df[bool_mask]` filters rows.
- Use `df.loc[row_mask, 'col'] = value` for conditional assignment — never chain `df[mask]['col'] = value` (SettingWithCopyWarning).
Every supervised ML dataset starts life as a DataFrame. `df[features]` (double brackets, list of column names) extracts the 2D `X` matrix; `df[target]` (single bracket) extracts the 1D `y` vector. `df.describe()` is the mandatory first step of any EDA. Filtering with boolean masks is how you split train/validation sets before `train_test_split`.
If you remove it: Without DataFrames, maintaining the correspondence between feature values and their column names would require a parallel list of strings — easily broken by sorting, filtering, or adding columns. Passing unlabelled arrays to scikit-learn is legal but removes all interpretability: `coef_[3]` means nothing; `coef_[df.columns[3]]` means 'the coefficient for age'. DataFrames keep metadata alive throughout the pipeline.