16 · Deep learning for sequences
RNNs and LSTMs read a series step by step and carry a hidden state — memory of the past. The setup that makes them learnable is windowing: slice the series into (input window, next value) pairs.
Recurrent networks (RNN/LSTM) process a sequence one step at a time, updating a hidden state that summarizes the past. You train them on windowed (input_window → next_value) supervised pairs — the same sliding window that any sequence model, classical or neural, consumes.
Without this:
Without windowing there's no supervised dataset to feed a neural net, and without recurrent state (or attention) a model can't carry long-range dependencies — both are prerequisites for deep sequence forecasting.
When the patterns get long and nonlinear, deep sequence models take over. A recurrent neural network (RNN) reads the series one step at a time, and at each step updates a hidden state h — a learned summary of everything seen so far. Feed it y[t] and h[t-1], and it produces h[t], which it can decode into a prediction. The hidden state is the model's memory.
Plain RNNs struggle to remember far back (vanishing gradients). The LSTM (Long Short-Term Memory) fixes this with gates — small learned valves (forget, input, output) over a separate cell state that lets information flow across many steps without decaying. That long memory is exactly what seasonal and multi-scale series need.
But before any neural net can learn, you need a supervised dataset, and the recipe is windowing (a.k.a. the sliding window). Pick a window length w. Slide it across the series: each window of w consecutive values is one input X, and the value right after it is the target y. A series of length n yields n - w training pairs. This is the same framing as the lag features from the previous lesson — just organized as fixed-length contiguous windows that an RNN consumes step by step.
Below we do the windowing in pure numpy so you can see the shapes exactly, then — since a real neural net needs PyTorch (not available here) — we stand in with a tiny linear autoregressive model fit by least squares (np.linalg.lstsq). It plays the role a one-layer network would: it learns weights mapping the window to the next value. We forecast a few steps and print predicted vs actual.
Python (in browser)
Windowing in pure numpy turns a series into (samples, window) X and next-value y; a least-squares linear AR model stands in for a neural net and forecasts recursively.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Here is the real thing as read-along code: a PyTorch LSTM forecaster. It cannot run in this browser sandbox (torch is not bundled in Pyodide), but every line is production-shaped and mirrors the numpy version above — same windowing, same time-ordered split, same recursive forecast. The only differences are the gated nn.LSTM cell carrying hidden + cell state across the window, a linear head decoding the final hidden state to one number, and an MSE-minimizing training loop with backpropagation through time. Read it as the destination the numpy stand-in was pointing toward.
READ-ALONG (torch not in Pyodide): a real PyTorch LSTM forecaster — windowed Dataset, nn.LSTM with a linear head, MSE training loop, and the same recursive multi-step rollout as the numpy stand-in.
What does the 'windowing' step produce, and why is it required before training an LSTM forecaster?
- RNNs/LSTMs read a sequence step by step and carry a hidden state — a learned memory of the past; LSTM gates let that memory survive across many steps.
- Windowing is the required setup: slide a length-w window to make (input_window → next_value) pairs, giving n - w supervised samples of shape (samples, window).
- A least-squares linear AR model is a zero-hidden-layer net on the same windowed data; an LSTM keeps the dataset and rollout identical but swaps in gated, nonlinear recurrence.
LSTMs and their attention-based successors (Temporal Fusion Transformer, time-series foundation models) drive forecasting where sequences are long, multivariate, and nonlinear — energy, traffic, sensor telemetry.
If you remove it: Without windowing you have no supervised dataset to train on, and without recurrent state (or attention) a model cannot carry long-range dependencies — deep sequence forecasting collapses to memoryless regression.