42 · Reading and writing data
Every ML project starts with pd.read_csv. Master the key options, understand JSON and Parquet alternatives, and learn the chunked-reader pattern for datasets that are larger than RAM.
Pandas is the **universal data importer**. One function — `pd.read_csv` — handles comma-separated, tab-separated, semicolon-separated, fixed-width, and gzip-compressed files. A second function — `pd.read_parquet` — handles the columnar binary format that is 3–10× smaller and 5–50× faster to read than CSV for analytical workloads. Understanding the `read_csv` options (`usecols`, `dtype`, `parse_dates`, `chunksize`) is not optional knowledge — it is how you control memory usage, parse times, and type correctness in every production pipeline.
Without this:
Without knowing `usecols`, loading a 50-column CSV to use 3 columns wastes 94% of the I/O. Without `dtype`, pandas infers types by scanning the entire file — slow and sometimes wrong (a zip-code column full of `'01234'` becomes an integer, losing the leading zero). Without `chunksize`, a 10 GB CSV crashes a 16 GB RAM machine. These are not edge cases — they are standard challenges in every real-world dataset.
Pandas ships a family of read_* functions covering almost every file format you will encounter in ML:
pd.read_csv(path)— the workhorse. Handles.csv,.tsv,.txt, even.csv.gz(gzip auto-detected from extension).pd.read_json(path)— JSON files. Useorient='records'for the most common layout (list of dicts).pd.read_parquet(path)— Parquet binary format. Requirespyarroworfastparquet. Typed, compressed, columnar — the production-grade successor to CSV.pd.read_excel(path)—.xlsx/.xls. Requiresopenpyxl.pd.read_sql(query, conn)— direct SQL query into a DataFrame.
Key read_csv options:
| Option | Purpose | Example |
|---|---|---|
| sep | delimiter character | sep='\t' for TSV |
| header | row number for column names | header=None if no header row |
| index_col | column to use as row index | index_col=0 |
| usecols | load only these columns | usecols=['age', 'income'] |
| dtype | override type inference | dtype={'zip': str} |
| nrows | load only the first N rows | nrows=1000 |
| parse_dates | parse these columns as datetime | parse_dates=['date'] |
| chunksize | return an iterator of DataFrames | chunksize=10_000 |
The mirror write function is df.to_csv(path, index=False). Pass index=False to avoid writing the row index as an extra column (the most common .to_csv mistake).
Python (in browser)
Pyodide runs in a virtual filesystem — `/tmp/` is writable and works exactly like a real filesystem for I/O demos. `df.equals(other)` checks shape, dtype, and every value — it is stricter than `==` on DataFrames (which returns an element-wise boolean DataFrame). Note that `to_csv(index=False)` is essential: without it, the saved file gains an unnamed first column of row numbers that becomes `Unnamed: 0` on read-back.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`usecols` skips parsing the unwanted columns entirely — for a 50-column CSV where you need 5, this is a 10× speedup and 10× memory reduction. `dtype={'zip_code': str}` prevents pandas from inferring `'01234'` as the integer `1234`. `parse_dates=['date']` converts the string column to `datetime64` so you can do date arithmetic immediately.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`orient='records'` is the most common JSON layout for ML data — each row is a dict, the whole file is a list: `[{'name': 'alice', 'score': 0.91}, ...]`. This is what most REST APIs and JavaScript front-ends produce. Other orients (`'split'`, `'index'`, `'columns'`, `'values'`) are used in specific contexts — `'split'` is space-efficient for sparse data.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The chunked reader pattern is how you train on datasets that don't fit in RAM. scikit-learn's `partial_fit` methods (SGDClassifier, SGDRegressor, MiniBatchKMeans) accept one chunk at a time — pair them with the chunked reader to process arbitrarily large datasets on a laptop. The chunk size (10,000 rows is a common default) is a tradeoff between memory usage and overhead per iteration.
What does `pd.read_csv(path, index_col=0)` do?
- `pd.read_csv(path)` is the universal entry point for tabular data. Always save with `to_csv(path, index=False)` to avoid a spurious `Unnamed: 0` column on read-back.
- Use `usecols` to load only needed columns, `dtype` to override type inference (especially for zip/ID columns), and `parse_dates` for date columns.
- `pd.read_csv(path, chunksize=N)` returns an iterator — use it to process datasets larger than RAM one batch at a time.
- Parquet (`to_parquet` / `read_parquet`) is smaller, typed, and faster than CSV for analytical queries. Use it for any dataset you will read more than once.
Every ML project's first line is `df = pd.read_csv('data.csv')` or `df = pd.read_parquet('data.parquet')`. scikit-learn's `fetch_*` utility functions return DataFrames internally. The chunked reader + `partial_fit` pattern is the standard approach for out-of-core (larger-than-RAM) model training. Kaggle competition data is almost always a `.csv` file read with `pd.read_csv`.
If you remove it: Without `read_csv` knowledge, loading data correctly (right types, right columns, no silent truncation) becomes trial-and-error for every new dataset. The `dtype` option alone prevents an entire category of bugs where numeric IDs get summed instead of used as keys, or leading-zero zip codes become wrong integers. These bugs are invisible until they surface as mysteriously wrong model predictions.