Abhinaash Blog

Why ordinary recurrent networks forget, the one equation that fixes it, and how gates turn that fix into a trainable memory cell. A mathematical walkthrough based on Hochreiter & Schmidhuber (1997) and Staudemeyer & Rothstein Morris (2019).

1. The sequence problem

A feed-forward network computes a static map $y = F(x)$. Many problems are not static: the right output at time $t$ depends on inputs seen at $t-1, t-2, \dots, t-1000$. Speech, handwriting, price series and text all have this shape. To model them we need a dynamic classifier, one with state that carries information forward.

Recurrent networks provide that state. In practice, however, a plain RNN trained by gradient descent can only exploit dependencies spanning roughly 5 to 10 steps. LSTM was designed to push that horizon beyond 1,000 steps. To see why it works, we first need to see precisely why the plain RNN fails.

2. RNNs and backpropagation through time

Take a fully recurrent network with non-input units $U$ and input units $I$. Each unit $u$ has activation function $f_u$, and $W_{[u,v]}$ is the weight from $v$ to $u$. The forward dynamics are

Over an epoch $[t', t]$ we minimise the summed squared error

Backpropagation through time (BPTT) unrolls the network into a deep feed-forward net with one layer per time step and shared weights. The error signal $\vartheta_u(\tau)$ obeys the backward recursion

and because weights are shared, their gradient is the sum over time:

Real-time recurrent learning (RTRL) computes the same gradient forwards instead, by carrying sensitivities $p^k_{uv}(\tau) = \partial y_k(\tau)/\partial W_{[u,v]}$:

BPTT needs memory proportional to sequence length; RTRL needs no history but costs $O(n^4)$ per step for $n$ units. The original LSTM uses a hybrid of the two, as we will see.

3. Why gradients vanish or explode

Hochreiter's 1991 analysis asks: if an error appears at unit $u$ at time $t$, how much of it reaches unit $v$ at time $t-q$? Differentiating the recursion above gives

Unrolling this by induction, with $l_0 = u$ and $l_q = v$:

This is a sum of $n^{q-1}$ paths, each a product of $q$ factors of the form $f'\cdot w$. The consequence is immediate:

• If $|f'_{l_m} w_{l_m l_{m-1}}| > 1$ for all $m$, the largest product grows like $c^q$. Errors explode and weights oscillate.
• If $|f'_{l_m} w_{l_m l_{m-1}}| < 1$ for all $m$, the product shrinks like $c^q$. Errors vanish and long-range learning stalls.

The sigmoid makes vanishing the default

For the logistic sigmoid $\sigma(x) = 1/(1+e^{-x})$ we have $\sigma'(x) = \sigma(x)(1-\sigma(x)) \le \tfrac14$. So $|f' w| < 1$ whenever $|w| < 4$, which covers nearly every weight at initialisation. Making weights larger does not help: for fixed input $y$, $|\sigma'(wy)\,w| \to 0$ as $|w|\to\infty$, because the derivative decays exponentially while the weight grows only linearly. Raising the learning rate does not help either, since it scales long- and short-range gradients equally.

A matrix bound

In matrix form, with $F'(t-m)$ the diagonal matrix of derivatives, the path sum equals

Using the max-row-sum norm $\|W\|_A = \max_r \sum_s |w_{rs}|$ and $f'_{\max} = \max_m \|F'(t-m)\|_A = \tfrac14$, one obtains

If every $|w_{ij}| \le w_{\max} < 4/n$, set $\rho = n w_{\max}/4 < 1$ and the bound becomes $n\rho^q$: guaranteed exponential decay. Summing over all output units shows that global error flow vanishes too. The same conclusion was reached independently by Bengio, Simard and Frasconi (1994).

4. The constant error carousel

The analysis tells us exactly what to demand. Consider a single unit $j$ with a self-connection $w_{jj}$. Its local backflow is

For the error to neither shrink nor grow we need

Integrating this differential equation gives $f_j(\text{net}_j) = \text{net}_j / w_{jj}$, so $f_j$ must be linear, and the activation must stay constant:

The simplest choice is $f_j = \mathrm{id}$ and $w_{jj} = 1$. This linear, self-connected unit is the constant error carousel (CEC), and it is the heart of LSTM: an error that enters it circulates unchanged for any number of steps.

5. Gates: the 1997 memory cell

LSTM wraps the CEC with two multiplicative gate units. An input gate $\text{in}_j$ decides when to write; an output gate $\text{out}_j$ decides when to read. The result is the memory cell $c_j$.

With all net inputs computed from the previous step's activations,

the cell updates its internal state and output as

The self-connection has weight exactly $1$, so the CEC is intact. The squashing functions used in the paper are

A memory cell block of size $S$ is a group of $S$ cells sharing one input gate and one output gate. It stores distributed information at the cost of only two gate units.

6. Truncated learning and exact error flow

The 1997 algorithm deliberately cuts gradient paths that leave a cell. It sets

Errors may still change the incoming weights, but they are not propagated further back through those connections. Only inside the cell do errors travel back through previous states. The immediate consequence is

Now trace an error $\vartheta_j(t)$ that arrives at the cell output and flows back $q$ steps to the cell input or the input gate:

Learning rules of the truncated algorithm

With squared error $E(t) = \sum_k (t^k(t) - y^k(t))^2$ and unit error $e_l(t) = -\partial E(t)/\partial\,\text{net}_l(t)$, the updates for output units, hidden units and output gates are ordinary backpropagation:

Input gates and cell inputs use an RTRL-style forward recursion. Define the internal-state error

and carry the derivatives of the state forward in time:

The weight updates are then $\Delta w_{\text{in}_j m}(t) = \alpha \sum_v e_{s_{c_j^v}}(t)\, \partial s_{c_j^v}(t)/\partial w_{\text{in}_j m}$ and $\Delta w_{c_j^v m}(t) = \alpha\, e_{s_{c_j^v}}(t)\, \partial s_{c_j^v}(t)/\partial w_{c_j^v m}$. Only these state derivatives need storing, so the algorithm is local in both space and time.

Two practical problems and their fixes

Abuse. Early in training the network can reduce error without memory, and may misuse cells as constant bias units. Fixes: add cells only when error stops falling, or give output gates negative initial biases (e.g. $-1, -2, -3$) so cells are "allocated" one at a time.

Internal state drift. If a cell's inputs are mostly of one sign, $s_{c_j}$ drifts, $h'(s_{c_j})\to 0$, and the gradient vanishes at the exit point. Fix: bias input gates negatively at initialisation (e.g. $-3, -6$).

7. Forget gates (Gers et al., 2000)

A fixed self-weight of $1$ has a flaw on continuous input streams: nothing ever resets the state. $s_{c}$ grows without bound, $h$ saturates, and the cell degenerates into an ordinary unit. Gers, Schmidhuber and Cummins replaced the constant $1$ with a learned forget gate $y^{\varphi}$:

The error through the carousel is now scaled by the forget gate each step, $\partial s(\tau+1)/\partial s(\tau) = y^{\varphi}(\tau+1)$. This is no longer exactly $1$, but it is a learned, input-dependent factor that the network can hold near $1$ for as long as it needs to remember and drop to $0$ when it wants to reset. The truncated state derivative for forget-gate weights is

8. The modern LSTM in vector form

Today's standard cell (as in PyTorch and TensorFlow) is the forget-gate LSTM with full backpropagation through time, $g=\tanh$, $h=\tanh$, and all four transforms computed from $[h_{t-1}, x_t]$. With hidden size $n$ and input size $d$:

Here $W_\ast \in \mathbb{R}^{n\times(n+d)}$, $b_\ast\in\mathbb{R}^n$, and $\odot$ is the element-wise product. The dictionary to the notation above is: $c_t \leftrightarrow s_c$, $\tilde c_t \leftrightarrow g(\text{net}_c)$, $i_t \leftrightarrow y^{\text{in}}$, $o_t\leftrightarrow y^{\text{out}}$, $f_t\leftrightarrow y^{\varphi}$, $h_t \leftrightarrow y^{c}$.

Two structural facts carry the whole design:

1. The cell update is additive. New information is added to $c_{t-1}$ rather than pushed through a matrix and a squashing function.
2. The direct Jacobian of the cell path is diagonal: $\dfrac{\partial c_t}{\partial c_{t-1}}\Big|_{\text{direct}} = \operatorname{diag}(f_t)$. Across $q$ steps this gives $\prod_{k=1}^{q}\operatorname{diag}(f_{t-k+1})$, with no weight matrices and no $\sigma'\le\tfrac14$ factors.

Some variants add peephole connections, letting gates see $c_{t-1}$ (or $c_t$ for the output gate), e.g. $f_t = \sigma(W_f[h_{t-1},x_t] + p_f\odot c_{t-1} + b_f)$. Gers et al. (2002) used these for precise timing tasks.

9. Backpropagation through the modern cell

Let $\delta h_t = \partial L/\partial h_t$ (loss gradient at step $t$ plus the gradient arriving from step $t+1$) and $\delta c_{t+1}$ be the gradient flowing back into the cell state from the future. Write $z_f, z_i, z_o, z_c$ for the pre-activations. Applying the chain rule to each line of the forward pass:

Stacking $\delta z_t = [\delta z_f;\delta z_i;\delta z_o;\delta z_c]$ and $W = [W_f;W_i;W_o;W_c]$, with $v_t = [h_{t-1},x_t]$:

Where the long-range gradient lives

Ignoring the paths through the gates, the gradient from $c_t$ back to $c_{t-q}$ is

With $f\equiv 1$ this is exactly the CEC. With $f = 0.99$ the signal after $100$ steps is still $0.99^{100}\approx 0.37$, while a plain RNN with factor $0.9$ retains $0.9^{100}\approx 2.7\times10^{-5}$. The chart at the top plots precisely this comparison. The full Jacobian also contains terms through $h_{t-1}$ and the gates; those behave like an ordinary RNN and do decay, which is why modern LSTMs are usually trained with gradient clipping.

10. A worked scalar example

Take $n=1$, $c_{t-1}=0.5$, and suppose the pre-activations come out as $z_f=1$, $z_i=0$, $z_o=2$, $z_c=1$.

Now backpropagate $\delta h_t = 1$ with $\delta c_{t+1}=0$:

Notice that $\delta c_{t-1}$ is simply $\delta c_t$ times the forget gate. No weight and no sigmoid derivative appears on that path.

11. A reference implementation

The NumPy code below implements Sections 8 and 9 directly. It passes a finite-difference gradient check with relative error around $4\times10^{-10}$.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def lstm_forward(X, h0, c0, W, b):
    """X: (T, d); W: (4n, n+d); b: (4n,). Gate order: f, i, o, g."""
    n = h0.shape[0]
    h, c, cache, H = h0, c0, [], []
    for x in X:
        v = np.concatenate([h, x])
        z = W @ v + b
        f, i, o = sigmoid(z[:n]), sigmoid(z[n:2*n]), sigmoid(z[2*n:3*n])
        g = np.tanh(z[3*n:])
        c_new = f * c + i * g
        tc = np.tanh(c_new)
        h_new = o * tc
        cache.append((v, c, f, i, o, g, tc))
        h, c = h_new, c_new
        H.append(h)
    return np.array(H), cache

def lstm_backward(dH, cache, W):
    """dH: (T, n) = dL/dh_t from the loss at each step."""
    n = dH.shape[1]
    dW, db = np.zeros_like(W), np.zeros(W.shape[0])
    dh_next, dc_next = np.zeros(n), np.zeros(n)
    for t in reversed(range(len(cache))):
        v, c_prev, f, i, o, g, tc = cache[t]
        dh = dH[t] + dh_next
        dc = dh * o * (1 - tc**2) + dc_next
        dz = np.concatenate([
            dc * c_prev * f * (1 - f),   # forget gate
            dc * g * i * (1 - i),        # input gate
            dh * tc * o * (1 - o),       # output gate
            dc * i * (1 - g**2),         # candidate
        ])
        dW += np.outer(dz, v)
        db += dz
        dh_next = (W.T @ dz)[:n]
        dc_next = dc * f                 # the carousel
    return dW, db

12. Cost, and the family of variants

Parameters and complexity

A modern LSTM layer has four weight matrices and four bias vectors:

For $n=128$, $d=10$ that is $4(128\cdot138+128) = 71{,}168$. For the original architecture, Hochreiter and Schmidhuber show the per-step update cost is

where $K$ is the number of outputs, $C$ the number of blocks of size $S$, $H$ conventional hidden units, $I$ the fan-in, and $W$ the number of weights. That is $O(1)$ per weight and time step, the same as BPTT, but without storing the activation history: only the $2CSI$ state derivatives $\partial s/\partial w$ are kept.

Gated recurrent unit (GRU)

Cho et al. (2014) merged the cell and hidden state and used two gates:

Comparing with $c_t = f_t\odot c_{t-1} + i_t\odot\tilde c_t$: the update gate $u_t$ plays the forget gate, and $1-u_t$ plays the input gate, coupling the two. GRU has three weight blocks instead of four and no output gate.

Architectural extensions

• Bidirectional LSTM runs one LSTM forwards and one backwards over the sequence and feeds both to the output layer, so each prediction sees past and future context. Combined with the CTC loss (Graves et al., 2006) it handles unsegmented sequences such as speech and handwriting.
• Stacked LSTM feeds layer $\ell-1$'s hidden state as layer $\ell$'s input: $X_\ell = [h^{(\ell-1)}_t;\ h^{(\ell)}_{t-1}]$.
• Multidimensional LSTM handles grid-structured data by giving each cell $N$ predecessors and $N$ forget gates: $s = \sum_{i=1}^N \varphi_i\odot s_i + \text{in}\odot z$.
• Grid LSTM (Kalchbrenner et al., 2016) applies a separate LSTM transform along each of $N$ dimensions, including depth, sharing the concatenated hidden vector but keeping distinct memory vectors per dimension. This avoids the instability of large multidimensional LSTMs.
• Sequence-to-sequence and attention. An encoder LSTM reads the input and a decoder LSTM generates the output (Sutskever et al., 2014); attention (Bahdanau et al., 2015) lets the decoder look back at all encoder states rather than a single fixed vector.

13. What the 1997 experiments showed

Every task except the first was built so that no short-lag training examples exist, and most were too hard for random weight guessing.

One scaling result stands out: when the number of distractor symbols grows in proportion to the time lag, training time grows very slowly, from 30,000 sequences at lag 50 to 49,000 at lag 1,000.

14. Limitations

• Non-decomposable tasks. Truncated LSTM struggles with strongly delayed XOR, where storing only one of the two inputs does not reduce error at all.
• Precise counting. Distinguishing whether a signal occurred 99 or 100 steps ago needs an extra mechanism; peephole connections help.
• Fixed capacity. The number of memory blocks is set in advance, so memory is bounded, and simply enlarging the network does not reliably fix that.
• Parameter cost. Each block adds gate units; in the fully connected case the weight count can rise by up to a factor of 9 relative to a plain RNN of the same hidden size.
• Sequential computation. The recurrence $c_t = f(c_{t-1}, \dots)$ cannot be parallelised across time, which is one reason attention-only models later displaced LSTMs for large-scale language tasks.

References

1. S. Hochreiter and J. Schmidhuber. Long Short-Term Memory. Neural Computation 9(8):1735–1780, 1997.
2. R. C. Staudemeyer and E. Rothstein Morris. Understanding LSTM: a tutorial into Long Short-Term Memory Recurrent Neural Networks. arXiv:1909.09586, 2019.
3. S. Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, TU München, 1991.
4. Y. Bengio, P. Simard, P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks 5(2), 1994.
5. F. A. Gers, J. Schmidhuber, F. Cummins. Learning to forget: continual prediction with LSTM. Neural Computation 12(10), 2000.
6. F. A. Gers, N. Schraudolph, J. Schmidhuber. Learning precise timing with LSTM recurrent networks. JMLR 3, 2002.
7. A. Graves et al. Connectionist temporal classification. ICML, 2006.
8. K. Cho et al. Learning phrase representations using RNN encoder–decoder. EMNLP, 2014.
9. R. Jozefowicz, W. Zaremba, I. Sutskever. An empirical exploration of recurrent network architectures. ICML, 2015.
10. N. Kalchbrenner, I. Danihelka, A. Graves. Grid Long Short-Term Memory. ICLR, 2016.