Abhinaash Blog

Why one function is reflected, why the other is shifted, and why the product is integrated — and how that single structure turns into smoothing, impulse responses, Green's functions and edge detectors.

Convolution appears almost everywhere in applied mathematics. It smooths noisy signals. It describes how physical systems respond to external inputs. It appears naturally in differential equations and Fourier analysis. In image processing it produces blurring, sharpening and edge detection. In machine learning, convolutional networks use a closely related operation to detect local patterns such as edges, textures and shapes.

Yet the definition

does not immediately reveal any of these ideas. Why is one function evaluated at $t$, while the other appears as $g(x-t)$? Why do we integrate their product? And why should this strange-looking operation be related to smoothing, pattern recognition, Fourier transforms and differential equations?

The key is to stop reading convolution as a formula and start reading it as a rule:

Convolution measures how information distributed across one function should influence another location, according to a translation-invariant weighting rule.

That single sentence connects nearly all of its seemingly different interpretations. The rest of this post is an unpacking of it.

1. Starting from the definition

For suitable functions $f, g : \mathbb{R} \to \mathbb{R}$, their convolution is

The result is another function. For every particular value of $x$ we compute one number, $(f*g)(x)$. The variable $t$ disappears after integration; it is only a dummy variable, like the index $i$ in a summation.

So $(f*g)(x)$ asks: how strongly do $f$ and a shifted version of $g$ interact when the output is evaluated at $x$?

The geometric interpretation comes from rewriting

Read as a function of $t$, this first reflects $g$ about the origin, $g(t) \mapsto g(-t)$, and then shifts the reflected function by $x$. That gives the familiar four-step recipe:

The graphical procedure is therefore not an arbitrary trick taught for computation. It is already encoded in the expression $g(x-t)$: the value $(f*g)(x)$ is the accumulated overlap between $f$ and a reflected, shifted copy of $g$.

2. Convolution as a weighted sum

There is a second reading that is often more useful than the flip-and-slide picture. Fix $x$ and look at the integrand:

For fixed $x$, the quantity $g(x-t)$ is simply a weight assigned to the value $f(t)$. Convolution is a weighted aggregation of $f$, and the kernel $g$ decides which locations matter.

Suppose $g$ is large when $t$ is close to $x$ and negligible when $t$ is far away. Then $(f*g)(x)$ depends almost entirely on values of $f$ near $x$, and convolution performs localised information aggregation. This is the viewpoint that carries over most directly to image filters and to convolutional networks.

Notice also what the weighting rule does not depend on: absolute position. The weight relating output location $x$ to input location $t$ is a function of the offset $x-t$ alone. The same rule is applied at every point. Section 14 shows that this property essentially characterises convolution.

3. Why convolution smooths functions

Take a Gaussian kernel with standard deviation $\sigma$,

and define $h = f * G_\sigma$, that is

Values $f(t)$ with $t \approx x$ receive large weights, because the exponential is largest there; values far from $x$ receive exponentially smaller weights. Because the kernel is non-negative and integrates to $1$, $h(x)$ is a genuine weighted average of $f$ near $x$ — in particular, constants are left unchanged.

Now suppose $f$ is a clean signal plus rapid noise. Neighbouring fluctuations of opposite sign are averaged against one another and partially cancel, while the slowly varying part survives almost untouched. The output is smoother. This is exactly the mechanism behind kernel smoothing in statistics and low-pass filtering in signal processing.

The regularising effect is not merely heuristic. Whenever the derivatives exist in a suitable sense, differentiation can be moved onto either factor:

The second equality is the useful one: to differentiate $f*g$ we may differentiate the kernel instead of $f$. So if $g$ is infinitely differentiable with compact support and $f$ is merely integrable, $f*g$ inherits the smoothness of $g$ and is itself infinitely differentiable. Convolving a rough function with a smooth kernel produces a smooth function, and this is one reason convolution is a basic tool in analysis and in partial differential equations.

4. A simple discrete example

Computers deal with sampled rather than continuous signals, and there the integral becomes a sum:

with values outside the stored sequence treated as zero. Take

indexed from $0$. Working through the sum term by term:

Therefore

The output has length $4 + 2 - 1 = 5$: convolution with a kernel of length $m$ extends a signal of length $n$ to length $n+m-1$. Away from the two boundary entries, the kernel $[1,1]$ is just adding neighbouring values. If instead we use

the same operation computes local averages — a moving-average filter, and the discrete counterpart of the Gaussian smoothing above. The weights summing to $1$ play the same role here as $\int G_\sigma = 1$ did there.

5. Convolution as a sum of shifted copies

A third interpretation follows from linearity, and it is probably the most useful mental model of all.

Suppose the input is a train of impulses,

understood in the distributional sense. Convolving with $g$ and using linearity,

and the defining property of the Dirac delta is that convolution with a shifted delta is a shift,

Hence

The output is a superposition of shifted, scaled copies of the kernel. In words:

Every point of the input launches its own translated copy of the kernel, scaled by the input's value at that point, and convolution adds all of those copies together.

This one picture explains smoothing, physical response systems, sums of independent random variables, and Green's functions using the same mathematics. It is also the picture that makes the next section almost inevitable.

6. Convolution and linear systems

Consider a system that accepts an input signal and returns an output. Feed it the simplest possible input, an impulse concentrated at the origin,

The resulting output $h$ is called the system's impulse response. Suppose the system has two properties:

• Linearity: the response to a sum of inputs is the sum of the responses.
• Time invariance: delaying the input only delays the output, so $\delta(t-\tau)$ produces $h(t-\tau)$.

Any signal can be written as a continuous superposition of impulses — this is the sifting property of the delta,

Linearity lets us respond to each impulse separately, and time invariance tells us what each response is. Summing them,

which is precisely

So for a linear time-invariant system,

The system is completely characterised by its response to a single impulse. This is one of the deepest reasons convolution appears so relentlessly in engineering and mathematical physics: it is not that convolution happens to be useful for linear systems, it is that linearity plus time invariance leaves no other possibility.

7. Fundamental algebraic properties

Convolution behaves remarkably like multiplication.

Commutativity

Substitute $u = x - t$, so that $t = x - u$ and $dt = -du$. As $t$ runs from $-\infty$ to $\infty$, $u$ runs from $\infty$ to $-\infty$; the sign from $dt$ cancels against the reversed limits, giving

This is worth pausing on. The flip-and-slide picture treats $f$ and $g$ asymmetrically — one stays put, the other is reflected and dragged across it — but the result does not care which one you chose to move. "Signal" and "kernel" is a distinction we impose, not one the operation makes.

Associativity, distributivity, scalars

Associativity has a concrete meaning: applying two filters in succession is the same as applying a single combined filter $g * h$, which is why cascaded blurs can be precomputed into one kernel.

The missing identity

One property of multiplication fails in an interesting way. There is no function $e \in L^1(\mathbb{R})$ with $e * f = f$ for all $f$. The identity element is the Dirac delta,

and $\delta$ is a distribution, not a function. This is not a technicality to be waved away — it is the reason approximate identities exist: kernels like $G_\sigma$ with $\sigma \to 0$ approach $\delta$ and recover $f$ in the limit, which is how smoothing is undone as the kernel narrows. It is also exactly what makes Section 10 work.

8. The Fourier transform changes everything

The resemblance to multiplication becomes an identity once we pass to the frequency domain.

The convolution theorem states that

Convolution in physical space becomes ordinary pointwise multiplication in frequency space. Here is why. Start with the definition and transform:

For $f, g \in L^1$ the double integral is absolutely convergent, so Fubini's theorem permits exchanging the order:

In the inner integral substitute $u = x - t$, so $x = u + t$ and $dx = du$:

The factor $\widehat{g}(\omega)$ no longer depends on $t$, so it comes out of the outer integral:

The entire proof is one substitution and one application of Fubini. The reason it works is visible in the middle step: shifting $g$ by $t$ multiplies its transform by $e^{-i\omega t}$, and convolution is built out of exactly those shifts. Complex exponentials turn translation into multiplication, and convolution is nothing but a weighted sum of translations.

9. Why smoothing makes sense in Fourier space

The convolution theorem gives a second, independent explanation of smoothing. If $h = f * g$, then

A smoothing kernel is one whose transform is close to $1$ at low frequencies and close to $0$ at high frequencies. The Gaussian is the model case: with the convention above,

which is again a Gaussian, but an inverted one — wide in space means narrow in frequency. Since $\widehat{G_\sigma}(\omega) \approx 0$ for large $|\omega|$, the product $\widehat{f}\widehat{g}$ suppresses precisely the high-frequency content of $f$, and a wider kernel (larger $\sigma$) suppresses more of it. Rapid fluctuations are high-frequency components, so the output is smoother.

The two explanations are the same statement in different coordinates:

Neither is more fundamental. The Fourier transform is simply the change of basis in which the operation becomes diagonal.

10. From convolution to differential equations

The same structure explains why convolution is entangled with differential equations. Consider

where $L$ is a linear differential operator with constant coefficients. Suppose we can find a $G$ satisfying

Such a $G$ is called a fundamental solution, or a Green's function. Now set $u = G * f$. Because $L$ has constant coefficients it commutes with translations, and therefore with convolution, so derivatives may be moved onto either factor:

The last step is the identity property from Section 7. Hence

Heat flow is the standard example. For $u_t = u_{xx}$ on the real line with initial data $u_0$, the solution is

Compare $\Phi$ with the Gaussian of Section 3: they match exactly when $\sigma = \sqrt{2t}$. Diffusion is Gaussian smoothing, with a kernel that widens as $\sqrt{t}$. The physical statement that heat spreads into neighbouring points and the analytic statement that convolution averages locally are the same statement, and the increasing width of the kernel is why detail is irreversibly lost as time runs forward.

11. Convolution as pattern matching

Now take a discrete local patch and a filter of the same size,

A local filtering operation computes

the inner product of the filter with the local data. Writing it geometrically,

the response is largest when $x$ points in the same direction as $w$. A filter therefore behaves like a local template matcher: if $w$ encodes a vertical edge, patches whose intensity pattern resembles a vertical edge produce a strong response. Slide the same filter across the image and the resulting map says where that pattern occurs. Weight sharing is what makes the answer meaningful — the same template is tested everywhere.

12. A subtle point about CNNs

There is a mathematical detail worth stating plainly. True discrete convolution reverses the kernel,

whereas the operation implemented by most deep-learning libraries is

with no reversal. The second is cross-correlation. The two are related by reflecting the kernel: writing $\tilde{g}(t) = g(-t)$,

so they carry the same information and differ only by which orientation of the kernel you call the kernel.

Does the distinction matter? Mathematically, yes: cross-correlation is not commutative and not associative, so the algebra of Section 7 does not transfer to it unchanged. Practically, for a network it does not: the filter weights are learned, so a network that would benefit from the reflected kernel simply learns the reflected weights. The two formulations describe the same family of functions. Calling the layer "convolution" is entrenched terminology rather than an error, but it is worth knowing which operation the code actually performs — it matters as soon as you hand-design a kernel, or compare against a signal-processing reference.

13. The deeper unifying picture

We can now put the readings side by side. Let $h = f * g$.

These are not seven tricks that happen to share a symbol. They are seven descriptions of one structure.

14. Why convolution appears everywhere

There is a final reason for the ubiquity, and it is the sharpest one: translation symmetry.

Suppose an operator $T$ acting on functions has two properties:

1. it is linear;
2. shifting the input merely shifts the output, $\;T[f(\cdot - a)](x) = (Tf)(x-a)$ for every $a$.

The second property says $T$ does not care about absolute location: it applies the same rule everywhere. Under suitable continuity assumptions, every such operator is a convolution,

for some kernel $k$. This is the real explanation, and it is stronger than any individual application: convolution is not a way to build translation-invariant linear operators, it is the only way.

This is also the point where this post meets the previous one. In Geometric Deep Learning, the same theorem appears in its discrete, finite form: a linear map on a cyclic grid commutes with the shift operator if and only if it is a convolution, which is why convolutional networks are forced by the demand for shift equivariance rather than chosen for convenience. Continuous or discrete, the content is identical. Ask for linearity and translation invariance, and convolution is what you get.

So whenever a system is linear and treats every location by the same rule — a lens, a filter, a homogeneous medium, a layer of a network — convolution is already there, whether or not anyone wrote the integral down.

Conclusion

At first sight

looks like a needlessly complicated integral. But every piece is doing a job. The $x - t$ supplies translation. The multiplication assigns influence. The integral aggregates every contribution. Together:

That single structure is why one operation can smooth a signal, model the response of a physical system, construct solutions of differential equations, detect patterns in images, and turn into multiplication under a Fourier transform. Perhaps the most revealing identity is therefore

because it shows that convolution is not an awkward alternative to multiplication. In the right representation, convolution is multiplication. And that is how an operation which first appears as a peculiar integral ended up as one of the organising ideas of modern applied mathematics.

Further reading

• Stein & Shakarchi. Fourier Analysis: An Introduction. Princeton Lectures in Analysis I, 2003. — convolution, approximate identities, the convolution theorem.
• Katznelson. An Introduction to Harmonic Analysis. — convolution as the multiplication of a Banach algebra.
• Bracewell. The Fourier Transform and Its Applications. — the graphical flip-and-slide picture, worked at length.
• Oppenheim & Willsky. Signals and Systems. — impulse responses and LTI systems.
• Evans. Partial Differential Equations. AMS Graduate Studies in Mathematics 19. — fundamental solutions and the heat kernel.
• Rudin. Functional Analysis. — distributions, tempered distributions, and translation-invariant operators.
• Goodfellow, Bengio, Courville. Deep Learning. MIT Press, 2016, Chapter 9. — the convolution versus cross-correlation convention in neural networks.