Abhinaash Blog

Every matrix is a rotation, a stretch, and another rotation. Once you see that, rank, conditioning, compression, least squares and PCA all turn out to be the same fact viewed from different angles.

A matrix can look like nothing more than a rectangular table of numbers. In applications, however, that table may represent an image, a set of user ratings, physical measurements, word frequencies, a linear transformation, or millions of parameters in a machine-learning model.

Singular Value Decomposition, usually called SVD, reveals the structure hidden inside that table. For every real $m \times n$ matrix $A$ it provides a factorisation

where $U$ and $V$ are orthogonal and $\Sigma$ holds nonnegative singular values. Unlike eigenvalue decomposition, this works even when $A$ is rectangular, and it works for every matrix without exception.

The importance of SVD is not that it rewrites one matrix as three. It is that the three factors expose the geometry, rank, dominant directions, null spaces, conditioning and information content of a linear transformation — all at once. That is why the same decomposition turns up in numerical analysis, optimisation, statistics, machine learning, signal processing, image compression, recommender systems and scientific computing.

1. What does SVD actually say?

Let $A \in \mathbb{R}^{m \times n}$. Then $A = U\Sigma V^{T}$, where

are orthogonal, meaning $U^{T}U = I_m$ and $V^{T}V = I_n$, and the middle factor $\Sigma \in \mathbb{R}^{m \times n}$ is diagonal in the rectangular sense:

with

The numbers $\sigma_1, \ldots, \sigma_r$ are the singular values of $A$. The zero blocks are whatever shape is needed to make $\Sigma$ an $m \times n$ matrix; if $A$ has full rank one or both of them may be absent.

The decomposition produces orthonormal bases adapted to the input and output spaces, and in those bases the transformation becomes purely diagonal. So the honest reading of $A = U\Sigma V^T$ is not "three matrices" but three actions:

2. The geometry behind the factorisation

Suppose a vector $x \in \mathbb{R}^n$ is transformed by $A$, giving $y = Ax$. Using the SVD, $y = U\Sigma V^{T}x$. Read the operations from right to left.

First,

Because $V$ is orthogonal, this changes the coordinate system without changing any lengths or angles. It is a rotation, possibly with a reflection. Nothing is distorted; we are only choosing a better set of axes to describe $x$.

Then,

This is the only step that does anything interesting. Each coordinate is independently scaled, with no mixing whatsoever. All of the transformation's distortion lives here, in $r$ numbers.

Finally,

another rigid change of coordinates, this time in the output space. Altogether:

Tracking a single basis vector through these three steps gives the equation that carries most of the meaning of SVD:

The vector $v_i$ is a special direction in the input space. The matrix $A$ sends it to the direction $u_i$ in the output space. Its length changes by exactly the factor $\sigma_i$. No rotation of $v_i$ into some unrelated direction, no shearing — along these particular directions, $A$ does nothing but scale.

3. The circle-to-ellipse picture

In two dimensions this becomes something you can draw. Consider every vector with $\|x\|_2 = 1$; these form the unit circle. Apply $A$ to all of them. The image is an ellipse.

The right singular vectors $v_1, v_2$ mark the special directions on the original circle. Their images are

so the resulting ellipse has principal semiaxes of length $\sigma_1$ and $\sigma_2$, pointing along $u_1$ and $u_2$. Singular values therefore have a completely concrete meaning:

(If $A$ is singular, some $\sigma_i$ vanishes and the ellipse degenerates — in two dimensions, collapsing to a line segment. That is the geometric face of rank deficiency.)

This explains why $\sigma_1$ is special. It is the largest stretching factor $A$ can produce anywhere:

and the maximising vector is $v_1$. The largest singular value is precisely the operator $2$-norm of $A$, which is why $\sigma_1$ shows up whenever someone needs to bound how much a matrix can amplify a vector.

4. Where do the singular values come from?

Why should such a factorisation exist at all? The answer starts with a trick: even when $A$ is rectangular, the matrix

is square and symmetric. It is also positive semidefinite, because for any $x$,

The spectral theorem therefore hands us an orthonormal basis of eigenvectors,

and since the eigenvalues cannot be negative we may take square roots:

This is one of the fundamental facts about SVD, and it is easy to see why it must hold. Substituting the factorisation,

which is an eigendecomposition of $A^{T}A$ with eigenvector matrix $V$ and eigenvalues $\sigma_i^2$. The right singular vectors are exactly the eigenvectors of $A^{T}A$, and the singular values are the square roots of its eigenvalues.

5. Constructing the left singular vectors

Once $v_i$ and $\sigma_i$ are known, define, for every nonzero singular value,

which immediately rearranges to $Av_i = \sigma_i u_i$. But we need these to form an orthonormal set, and they do. For $i \neq j$,

using $A^{T}Av_j = \sigma_j^2 v_j$ in the third step and the orthogonality of the $v_i$ in the last. The same computation with $i = j$ shows they are unit vectors,

so the construction is complete:

Orthonormality of the left singular vectors is not an extra assumption we impose. It falls out of the orthogonality of the right singular vectors, for free. SVD naturally produces two orthonormal coordinate systems, one on each side.

When $r < m$, the vectors $u_1, \ldots, u_r$ do not yet fill $\mathbb{R}^m$; the remaining columns of $U$ are obtained by extending them to an orthonormal basis, and Section 7 says what those extra directions mean.

6. Why two sets of singular vectors?

Eigenvalue decomposition looks for vectors satisfying

which asks $A$ to map a direction back onto itself. That demand only makes sense when $A$ maps a space to itself. If

then $Ax$ and $\lambda x$ live in different spaces and the equation does not even typecheck. Eigenvectors of a rectangular matrix are not a hard problem; they are a meaningless one.

SVD sidesteps this by allowing a different basis on each side:

Instead of asking which directions are preserved, SVD asks which input directions go to which output directions, and by how much. That question always has an answer. This is also why SVD avoids the other failures of eigendecomposition: eigenvectors need not be orthogonal, a matrix may not have enough of them to form a basis, and the eigenvalues may be complex even for a real matrix. Singular values are real, nonnegative, and always exist.

7. SVD reveals the four fundamental subspaces

Suppose $\operatorname{rank}(A) = r$. Then the singular vectors hand you all four fundamental subspaces, already equipped with orthonormal bases. On the output side,

and on the input side,

The split is exactly where the singular values run out. Directions with $\sigma_i > 0$ survive the transformation; directions past the $r$th are annihilated. Reading off the rank is then trivial — it is the number of nonzero singular values — and the orthogonal complements come for free, since the columns of $U$ and of $V$ were orthonormal to begin with.

This is a good reason to regard SVD as more than a computational device. It reorganises the entire subspace structure of $A$ into a single picture.

8. SVD as a sum of rank-one matrices

Multiplying out $U\Sigma V^{T}$ column by column gives a different and very useful form:

Each outer product $u_i v_i^{T}$ is an $m \times n$ matrix of rank one — about as simple as a matrix can be without being zero. So SVD expresses an arbitrary rank-$r$ matrix as a sum of exactly $r$ extremely simple matrices, and the sum is ordered: the terms come with decreasing weights $\sigma_1 \geq \sigma_2 \geq \cdots$.

That ordering is what makes the expansion valuable:

where the importance of the $i$th pattern is measured by $\sigma_i$. The matrix has been decomposed into layers of structure, strongest first.

9. The first singular component

The leading term $A_1 = \sigma_1 u_1 v_1^{T}$ captures the strongest rank-one structure inside $A$. Adding the next term,

captures the two strongest components, and in general we truncate after $k$ terms:

This is the truncated SVD. The remarkable part is not that it is a convenient approximation. It is that among all rank-$k$ matrices, it is the best one.

10. The Eckart–Young–Mirsky theorem

Suppose we want the closest rank-$k$ approximation to $A$. There are infinitely many rank-$k$ matrices to choose from, and no obvious reason the answer should be simple. But it is: among every matrix $B$ with $\operatorname{rank}(B) \leq k$, the truncated SVD $A_k$ minimises the error. In the Frobenius norm,

and in the spectral $2$-norm,

This is the Eckart–Young–Mirsky theorem, and it is one of the deepest practical reasons for the success of SVD. Note what it says about the error: it is written entirely in terms of the singular values you threw away. Before computing any approximation, the spectrum already tells you how good it will be.

The consequence is the one everyone uses. If $\sigma_1, \sigma_2, \sigma_3$ are large while $\sigma_4, \sigma_5, \ldots$ are tiny, then essentially all of the information in $A$ lives in three directions. Rather than storing the whole matrix, keep

and discard the rest. This is dimensionality reduction in its most fundamental linear-algebraic form, and the theorem certifies that nothing better exists at that rank.

11. Why image compression works

Consider a grayscale image stored as $A \in \mathbb{R}^{m \times n}$, where each entry $a_{ij}$ is a pixel intensity, and write its SVD as before. A photograph usually contains strong spatial redundancy: neighbouring pixels are related, large regions are nearly uniform, structures repeat. That redundancy shows up as rapidly decaying singular values,

which is exactly the situation in which truncation is cheap. Keeping $k$ terms means storing $k$ vectors of length $m$, $k$ vectors of length $n$, and $k$ singular values:

The approximation is therefore worth doing whenever

which for a square image of side $n$ is roughly $k < n/2$. In practice useful reconstructions often need far fewer terms than that, so the saving can be substantial — and the Eckart–Young–Mirsky theorem guarantees that for the storage budget spent, no rank-$k$ scheme reproduces the image more faithfully.

12. SVD and noise

Suppose

where $E$ represents noise. If the structure in $A_{\text{signal}}$ is approximately low rank, it will tend to concentrate in the first few singular values, while noise — having no preferred direction — spreads itself thinly across all of them. Truncating therefore acts as a form of regularisation: we deliberately discard the weak directions, keeping $A_k$ and throwing away what is left.

It is worth being careful here. "Small singular value" and "noise" are not synonyms; whether the identification is justified depends on the data and on the modelling assumptions, and a genuinely small but real effect will be discarded along with the noise. But as a working principle, in problems where the signal really is low rank and the noise really is diffuse, it is remarkably effective.

13. Numerical rank

Exact mathematical rank is a discrete quantity, and that makes it fragile. Consider a matrix whose singular values are

All four are nonzero, so $\operatorname{rank}(A) = 4$, exactly. Yet the matrix behaves for every practical purpose like a rank-three matrix, and the tiny fourth direction is more likely to be rounding error or measurement noise than real structure.

Worse, rank computed by exact arithmetic is discontinuous: an arbitrarily small perturbation can turn a rank-three matrix into a full-rank one. SVD replaces this brittle notion with a stable one,

treating singular values below a tolerance as zero. Because Eckart–Young–Mirsky tells us $\sigma_{k+1}$ is exactly the distance from $A$ to the nearest rank-$k$ matrix, this is not an arbitrary convention: it says the matrix is within $\text{tol}$ of one with that exact rank. SVD is the standard tool for detecting near-dependencies precisely because it measures how close to singular a matrix is, not merely whether it is.

14. SVD and the Moore–Penrose pseudoinverse

Suppose we want to solve $Ax = b$. If $A$ is square and invertible, $x = A^{-1}b$ and there is nothing to discuss. Real problems are rarely so accommodating: $A$ may be rectangular, rank-deficient, or both, in which case the system may have no solution or infinitely many.

SVD gives the natural generalisation of the inverse. Starting from $A = U\Sigma V^{T}$, define $\Sigma^{\dagger} \in \mathbb{R}^{n \times m}$ by transposing $\Sigma$ and replacing each nonzero $\sigma_i$ with $1/\sigma_i$, leaving the zeros alone. Then

is the Moore–Penrose pseudoinverse, and

is the minimum-norm least-squares solution. That single description covers both awkward cases at once: among all $x$ minimising $\|Ax - b\|_2$, it returns the one of smallest norm. If the system is overdetermined and inconsistent, the least-squares part does the work; if it is underdetermined with infinitely many exact solutions, the minimum-norm part picks one canonically.

The geometry is transparent in the SVD basis. Applying $U^{T}$ expresses $b$ in the output basis; the nonzero part of $\Sigma^{\dagger}$ undoes the stretching in the $r$ directions where $A$ actually acted; the zeros ignore the components of $b$ lying in $N(A^{T})$, which no $x$ could have produced, and refuse to assign anything to $N(A)$, which is what keeps the answer's norm minimal. Inverting where inversion makes sense, and projecting away where it does not.

15. Condition numbers and ill-conditioning

Suppose $A$ is square and of full rank. Its spectral condition number is

If $\sigma_r$ is extremely small then $1/\sigma_r$ is extremely large, and small perturbations in the data get amplified by that factor when solving $Ax = b$. The problem is ill-conditioned — not because the algorithm is bad, but because the matrix itself destroys information along one direction, and no algorithm can recover what the matrix discarded.

Geometrically this is the circle-to-ellipse picture again, in a degenerate regime. The transformation maps the unit sphere to a very thin ellipsoid: one direction stretched a great deal, another nearly flattened. Inverting means re-inflating the flattened direction, which magnifies whatever error happened to lie there. So the condition number measures anisotropy:

A perfectly conditioned matrix has $\kappa_2 = 1$ and stretches every direction equally — a scaled orthogonal matrix, which maps the sphere to a sphere.

16. SVD versus eigenvalue decomposition

The two decompositions answer different questions. Eigendecomposition $A = Q\Lambda Q^{-1}$ describes invariant directions, $Av_i = \lambda_i v_i$: feed in $v_i$ and the same direction comes back. SVD describes input-output directions, $Av_i = \sigma_i u_i$: feed in $v_i$ and a possibly different direction comes out.

For a symmetric matrix the two nearly coincide, and the relationship is exactly

with the singular vectors equal to the eigenvectors up to sign — a negative eigenvalue becomes a positive singular value with the corresponding $u_i$ flipped, since SVD records stretching in a magnitude and puts the direction reversal into $U$.

One more connection worth knowing: since $|\det U| = |\det V| = 1$ for orthogonal matrices, a square matrix satisfies

so the determinant measures volume scaling, which is just the product of the stretching factors along the principal axes. A single small singular value drags the determinant toward zero, which is the algebraic shadow of the flattened ellipsoid from the previous section.

17. SVD and principal component analysis

Let a centred dataset be $X \in \mathbb{R}^{m \times n}$, with $m$ samples in rows and $n$ features in columns, each column having mean zero. Take its SVD, $X = U\Sigma V^{T}$. Then

and the sample covariance matrix is

That is already an eigendecomposition of $C$, so we can read the answer off directly: the principal directions are the columns of $V$, and the variance captured by the $i$th is $\sigma_i^2/(m-1)$.

So PCA is not merely related to SVD; PCA is SVD applied to centred data. This is also how PCA is computed in practice, since taking the SVD of $X$ avoids ever forming $X^{T}X$ and the loss of accuracy that comes with it. The centring genuinely matters: SVD of uncentred data gives directions of largest second moment rather than largest variance, and the leading direction usually just points at the mean.

18. SVD and recommendation systems

Imagine a matrix $R$ whose rows are users and whose columns are movies, with $r_{ij}$ the rating user $i$ gave movie $j$. The raw matrix may be enormous, but preferences plausibly depend on far fewer underlying factors — a taste for action, for comedy, for slow character studies. If so, $R$ is approximately low rank, and a truncated factorisation

exposes that latent structure. The columns of $U_k$ describe user-side latent directions, the columns of $V_k$ describe item-side latent directions, and the singular values say how strong each factor is. Two users with similar coordinates in the latent space have similar taste, even if they have never rated the same film.

19. A small worked example

Take

First compute

whose eigenvalues are $25 \pm 20$, namely $45$ and $5$. The singular values are their square roots:

The corresponding normalised eigenvectors are

Now build the left singular vectors from $u_i = Av_i/\sigma_i$. For the first,

and for the second,

Assembling the three factors,

and multiplying back confirms it:

As a second check, $\det A = 15$ and $\sigma_1\sigma_2 = 3\sqrt{5}\cdot\sqrt{5} = 15$, matching the determinant identity from Section 16.

But the numbers matter less than what they say. The direction $v_1$, running diagonally at $45^\circ$, is stretched by $\sqrt{45} \approx 6.7$, while the perpendicular direction $v_2$ is stretched only by $\sqrt{5} \approx 2.2$. The matrix looked like four arbitrary entries; SVD has extracted the two natural axes along which it acts, and the factor by which it acts on each.

20. The deeper idea

SVD is usually introduced as a formula, $A = U\Sigma V^{T}$. But its real content is about choosing the right coordinates.

In the standard basis, the action of $A$ looks complicated. It mixes coordinates: every output entry depends on every input entry, and there is no obvious structure in the table of numbers. SVD finds special orthonormal bases — one for the input space, one for the output space — in which that same transformation becomes just $\Sigma$. And $\Sigma$ does nothing mysterious at all. It says

The transformation has been reduced to a list of independent stretches. Nothing was approximated and nothing was lost; the complexity was an artefact of describing the map in the wrong basis. So

should be read as the statement that a complicated linear transformation is simple when viewed from the correct input and output bases — and that for a linear map, such bases always exist.

Conclusion

Singular Value Decomposition is one of the central organising ideas of linear algebra because a single computation answers so many separate questions at once. Given a matrix $A$, its SVD tells us:

At the geometric level, SVD finds the natural axes of a linear transformation. At the algebraic level, it writes that transformation as

At the level of data, it ranks hidden patterns by importance. And computationally, it underwrites compression, regularisation, dimensionality reduction, least squares, PCA, denoising and latent-factor discovery — all of which are the same theorem wearing different clothes.

Perhaps the most useful way to remember it is therefore not the formula but the sentence behind it:

That is the hidden geometry SVD extracts from a matrix — and the reason a table of numbers is never just a table of numbers.

Further reading

• Trefethen & Bau. Numerical Linear Algebra. SIAM, 1997, Lectures 4–5. — the geometric derivation, done properly.
• Golub & Van Loan. Matrix Computations. Johns Hopkins University Press. — the standard reference on computing the SVD stably.
• Strang. Linear Algebra and Learning from Data. Wellesley–Cambridge Press, 2019. — SVD, low-rank approximation and PCA from an applied angle.
• Horn & Johnson. Matrix Analysis. Cambridge University Press. — singular value inequalities and the underlying theory.
• Eckart & Young. The approximation of one matrix by another of lower rank. Psychometrika, 1936. — the original low-rank approximation result.
• Mirsky. Symmetric gauge functions and unitarily invariant norms. Quarterly Journal of Mathematics, 1960. — the generalisation to all unitarily invariant norms.
• IBM. What Is Singular Value Decomposition? ibm.com/think/topics/singular-value-decomposition