8 · Case study: Flight Price feature engineering
Parse dates and durations, encode airlines, and log-transform a skewed price target — the full feature engineering pipeline for a flight-price regression problem.
EDA is half the work — the other half is feature engineering: parsing dates, extracting durations, encoding airline names, and creating ratios.
Without this:
Without feature engineering, your raw 'date of journey: 24/03/2019' column is useless to a numeric model.
The wine-quality dataset was clean and numeric — a best-case EDA. Real Kaggle datasets look nothing like that.
The Flight Price dataset is a common first encounter with the messiness of real-world tabular data:
- Dates arrive as strings:
"24/03/2019" - Durations arrive as strings:
"2h 50m","1h 5m" - Stop counts arrive as strings:
"non-stop","1 stop","2 stops" - The price target is heavily right-skewed
- Airline names are nominal with moderate cardinality (~5 carriers)
Feature engineering converts these unusable strings into the numeric columns a model needs. The output of this lesson is a model-ready X matrix with columns like journey_day, duration_min, total_stops, airline_freq — and a log_price target.
Python (in browser)
The raw DataFrame exposes the parsing challenge: `date_of_journey`, `duration`, and `total_stops` are all `object` dtype. `df.dtypes` is the first diagnostic — it shows you which columns need engineering before any model can use them.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`pd.to_datetime(..., format='%d/%m/%Y')` parses the string column; `.dt.day`, `.dt.month`, `.dt.dayofweek` extract the numeric components. Weekend = `dayofweek >= 5` — a binary feature that often predicts price bumps.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
A small `to_minutes` function splits on 'h' and 'm' to extract hours and minutes from strings like `'2h 50m'`. The resulting `duration_min` is a single, model-ready continuous feature.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`stops_map = {'non-stop': 0, '1 stop': 1, '2 stops': 2}` — a pure pandas `.map()` converts the ordinal string to an integer in one line. Stops are genuinely ordinal (more stops = longer journey = usually higher price), so an integer code is appropriate here.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
`np.log1p(price)` compresses the right tail and reduces skewness from ~2 toward 0. The transformed target is much closer to Normal — a requirement for well-calibrated linear regression residuals.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Frequency encoding airline with `df['airline'].map(df['airline'].value_counts())` — one line, no leakage risk, captures carrier market share. Useful when carrier popularity is itself a price predictor (budget vs premium airlines).
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The post-feature-engineering pipeline: raw columns become numeric X columns, log-transformed price becomes y. Note `np.expm1()` to invert the log transform at prediction time.
Why log-transform the price target before fitting a linear regression?
- `pd.to_datetime(..., format='...')` + `.dt.day` / `.dt.month` / `.dt.dayofweek` converts a date string into multiple numeric features — day, month, weekday, is_weekend.
- Duration strings like `'2h 50m'` require a small parsing function that splits on `'h'` and `'m'` to produce a single `duration_min` integer — a pattern that recurs in every logistics dataset.
- For a right-skewed numeric target (flight price, house price, salary), apply `np.log1p(y)` before fitting and `np.expm1(y_pred)` to recover the original scale after prediction.
- Feature engineering on tabular data consistently delivers more improvement than switching models — prioritise engineered features over model architecture.
Real ML pipelines spend 60-80% of engineer time on EDA + feature engineering. Tabular ML competitions on Kaggle are won by feature engineers, not model tinkerers.
If you remove it: Raw string columns are useless — your model.fit() either crashes or treats hex codes as ordinal.