I have seen the Singular Value Decomposition (SVD) represented in two different ways and wanted to know a) if they're both correct and b) what their relationship is.
In the first version a matrix $A \in \mathbb{R}^{mxn}$ such that $rank(A)=s$, can be decomposed as $A=VDU^t$ where $V \in \mathbb{R}^{mxs}$, $D \in \mathbb{R}^{sxs}$ and $U \in \mathbb{R}^{nxs}$.
In the second version a matrix $A \in \mathbb{R}^{mxn}$, can be decomposed as $A=VDU^t$ where $V \in \mathbb{R}^{mxm}$, $D \in \mathbb{R}^{mxn}$ and $U \in \mathbb{R}^{nxn}$.
$\endgroup$ 12 Answers
$\begingroup$Both representations are correct, and they coincide as far as the smaller matrices $U,V$. The first one is called "thin SVD". The second one has the same elements of $D$ as the first one, but the rest of the matrix is filled out with zeros.
The difference is that matrices $U$ and $V$ in the "larger" representation contain extra columns, which annihilate after being multiplied by zero elements of $D$.
$\endgroup$ $\begingroup$Start with the matrix $$ \mathbf{A} \in \mathbb{C}^{m\times n}_{\rho} $$ and consider the most general case of rank deficiency where $\rho < \min \left( m, n\right)$. The singular value decomposition is $$ \begin{align} \mathbf{A} &= \mathbf{U} \, \Sigma \, \mathbf{V}^{*} \\ % &= % U \left[ \begin{array}{cc} \color{blue}{\mathbf{U}_{\mathcal{R}}} & \color{red}{\mathbf{U}_{\mathcal{N}}} \end{array} \right] % Sigma \left[ \begin{array}{cccc|cc} \sigma_{1} & 0 & \dots & & & \dots & 0 \\ 0 & \sigma_{2} \\ \vdots && \ddots \\ & & & \sigma_{\rho} \\\hline & & & & 0 & \\ \vdots &&&&&\ddots \\ 0 & & & & & & 0 \\ \end{array} \right] % V \left[ \begin{array}{c} \color{blue}{\mathbf{V}_{\mathcal{R}}}^{*} \\ \color{red}{\mathbf{V}_{\mathcal{N}}}^{*} \end{array} \right] \\ % & = % U \left[ \begin{array}{cccccccc} \color{blue}{u_{1}} & \dots & \color{blue}{u_{\rho}} & \color{red}{u_{\rho+1}} & \dots & \color{red}{u_{n}} \end{array} \right] % Sigma \left[ \begin{array}{cc} \mathbf{S}_{\rho\times \rho} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} \end{array} \right] % V \left[ \begin{array}{c} \color{blue}{v_{1}^{*}} \\ \vdots \\ \color{blue}{v_{\rho}^{*}} \\ \color{red}{v_{\rho+1}^{*}} \\ \vdots \\ \color{red}{v_{n}^{*}} \end{array} \right] % \end{align} $$ This presents orthonormal bases for both the domain and the codomain. The Fundamental Theorem of Linear Algebra prescribed four fundamental subspaces:
$$ \begin{align} % \mathbf{C}^{m} = \color{blue}{\mathcal{R} \left( \mathbf{A} \right)} \oplus \color{red} {\mathcal{N} \left( \mathbf{A}^{*} \right)} \\ % \mathbf{C}^{n} = \color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)} \oplus \color{red} {\mathcal{N} \left( \mathbf{A} \right)} % \end{align} $$
$$ \begin{array}{ll} % column \ vectors & span \\\hline % \color{blue}{u_{1}} \dots \color{blue}{u_{\rho}} & \color{blue}{\mathcal{R} \left( \mathbf{A} \right)} \\ % \color{blue}{v_{1}} \dots \color{blue}{v_{\rho}} & \color{blue}{\mathcal{R} \left( \mathbf{A}^{*} \right)} \\ % \color{red}{u_{\rho+1}} \dots \color{red}{u_{m}} & \color{red}{\mathcal{N} \left( \mathbf{A}^{*} \right)} \\ % \color{red}{v_{\rho+1}} \dots \color{red}{v_{n}} & \color{red}{\mathcal{N} \left( \mathbf{A} \right)} \\ % \end{array} $$
The SVD can be expressed solely in terms of range space quantities. This is the thin or economical SVD $$ \mathbf{A} = \color{blue}{\mathbf{U}_{\mathcal{R}}} \, \mathbf{S} \, \color{blue}{\mathbf{V}_{\mathcal{R}}}^{*} $$ Write the dimensions explicitly $$ \begin{array}{ccccc} % \mathbf{A} & = & \color{blue}{\mathbf{U}_{\mathcal{R}}} & \mathbf{S} & \color{blue}{\mathbf{V}_{\mathcal{R}}}^{*} \\ % m \times n & & m \times \rho & \rho \times \rho & \rho \times n \\[5pt] % \mathbf{A} & = % & \mathbf{U} & \Sigma & \mathbf{V}^{*} \\ % m \times n & & m \times m & m \times n & n \times n \\ % \end{array} $$
$\endgroup$
