9 · Case study: Google Play Store data cleaning
Tackle the string-typed numerics, mixed units, and duplicates that make `df.describe()` useless — and build a fully numeric, analysis-ready DataFrame.
Real-world CSVs are filthy — string-typed numerics, inconsistent units, parse-resistant categoricals. Cleaning is half of EDA.
Without this:
Without cleaning, `df.describe()` gives nothing on the 'Size' column because it's stored as '19M', '85k', 'Varies with device'.
Every Kaggle newbie has a moment where they download a "real" CSV, run df.describe(), and see almost nothing useful — because half the numeric columns are stored as object dtype.
The Google Play Store dataset is the lesson that humbled most data science beginners. Its Size column contains values like '19M', '85k', and 'Varies with device'. Its Installs column reads '1,000,000+'. Its Price column stores '$4.99' for paid apps and '0' for free.
None of these are numeric. None are directly usable. This lesson shows the full data cleaning pipeline:
- Detect type issues with
df.dtypes - Parse string-typed numerics column by column
- Standardise units (M → bytes, k → bytes)
- Handle sentinel values (
'Varies with device'→NaN) - Drop exact duplicates
- Save the cleaned DataFrame
The key discipline: never mutate the original DataFrame during cleaning. Always create new columns or work on a df.copy().
Python (in browser)
`df.dtypes` is the first diagnostic — it reveals that `Size`, `Installs`, and `Price` are `object` when they should be numeric. `df.describe()` on these columns produces a categorical summary (count/unique/top/freq) rather than the five-number summary. These are the columns that need cleaning.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`parse_size` handles all three cases: 'M' (megabytes → bytes), 'k' (kilobytes → bytes), 'Varies with device' → `NaN`. Applying it with `.apply()` creates a clean `size_bytes` column without touching the original `Size`.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`.str.replace(',','').str.replace('+','').astype(int)` — a three-step chain that strips formatting characters and casts to int. This pattern generalises to any 'number-with-suffix' string column.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`.str.replace('$','').astype(float)` strips the currency symbol and casts to float. `drop_duplicates(subset=['App'], keep='first')` removes exact duplicate rows — always specify `subset` so you only dedup on the key columns you care about.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The six-step cleaning checklist as a reusable code comment. Each step is a distinct, reversible transformation applied to new columns — never the originals. Deterministic + re-runnable = production-ready ETL.
An 'Installs' column has the values `'1,000,000+'` and `'10,000+'`. What's the simplest pandas chain to convert both to integers?
- `df.dtypes` is the cleaning entry point — any numeric column stored as `object` needs parsing before analysis or modelling.
- The `.str` accessor chains — `.str.replace().str.replace().astype()` — are the standard pandas pattern for cleaning formatted strings like `'1,000,000+'` or `'$4.99'`.
- Sentinel strings (`'Varies with device'`, `'Unknown'`, `'N/A'`) should be mapped to `np.nan` so downstream imputers and `.fillna()` can handle them uniformly.
- Never mutate original columns during cleaning — create new columns and keep the raw ones as an audit trail. The cleaning chain must be deterministic and re-runnable.
Web scrapes, API dumps, and CRM exports are 'object'-typed across half their columns; cleaning is the gate every data scientist passes through. ETL pipelines codify the cleaning logic so the same raw → clean transformation runs nightly.
If you remove it: You'd train a model on the literal string 'Varies with device' and quietly produce garbage.