Questions tagged [svd]

In linear algebra, the singular value decomposition (SVD) is a factorization of a real or complex matrix, with many useful applications in signal processing and statistics.

In linear algebra, the singular-value decomposition (SVD) is a factorization of a real or complex matrix.

Formally, the singular value decomposition of an $m \times n$ real or complex matrix $M$ is a factorisation of the form $UAV^*$ where $U$ is an $m\times m$ real or complex unitary matrix, $A$ is an $m\times n$ rectangular diagonal matrix with non-negative real numbers on the diagonal, and $V$ is an $n\times n$ real or complex unitary matrix.

The singular-value decomposition can be computed using the following observations:

  • The left-singular vectors of $M$ are a set of orthonormal eigenvectors of $MM^*$.
  • The right-singular vectors of $M$ are a set of orthonormal eigenvectors of $M^*M$.
  • The non-zero singular values of $M$ (found on the diagonal entries of $A$) are the square roots of the non-zero eigenvalues of both $M^*M$ and $MM^*$.

Source: Wikipedia.

1355 questions
439
votes
4 answers

What is the intuitive relationship between SVD and PCA?

Singular value decomposition (SVD) and principal component analysis (PCA) are two eigenvalue methods used to reduce a high-dimensional data set into fewer dimensions while retaining important information. Online articles say that these methods are…
164
votes
8 answers

What is the difference between "singular value" and "eigenvalue"?

I am trying to prove some statements about singular value decomposition, but I am not sure what the difference between singular value and eigenvalue is. Is "singular value" just another name for eigenvalue?
161
votes
7 answers

Intuitively, what is the difference between Eigendecomposition and Singular Value Decomposition?

I'm trying to intuitively understand the difference between SVD and eigendecomposition. From my understanding, eigendecomposition seeks to describe a linear transformation as a sequence of three basic operations ($P^{-1}DP$) on a vector: Rotation…
user541686
  • 12,494
  • 15
  • 48
  • 93
60
votes
2 answers

What do eigenvalues have to do with pictures?

I am trying to write a program that will perform OCR on a mobile phone, and I recently encountered this article : Can someone explain this to me ?
55
votes
3 answers

How unique are $U$ and $V$ in the Singular Value Decomposition?

According to Wikipedia: A common convention is to list the singular values in descending order. In this case, the diagonal matrix $\Sigma$ is uniquely determined by $M$ (though the matrices $U$ and $V$ are not). My question is, are $U$ and $V$…
46
votes
2 answers

Why does the spectral norm equal the largest singular value?

This may be a trivial question yet I was unable to find an answer: $$\left \| A \right \| _2=\sqrt{\lambda_{\text{max}}(A^{^*}A)}=\sigma_{\text{max}}(A)$$ where the spectral norm $\left \| A \right \| _2$ of a complex matrix $A$ is defined as…
mathemage
  • 670
  • 1
  • 6
  • 13
39
votes
6 answers

How can you explain the Singular Value Decomposition to non-specialists?

In two days, I am giving a presentation about a search engine I have been making the past summer. My research involved the use of singular value decompositions, i.e., $A = U \Sigma V^T$. I took a high school course on Linear Algebra last year, but…
38
votes
1 answer

Relationship between eigendecomposition and singular value decomposition

Let $A \in \mathbb{R}^{n\times n}$ be a real symmetric matrix. Please help me clear up some confusion about the relationship between the singular value decomposition of $A$ and the eigen-decomposition of $A$. Let $A = U\Sigma V^T$ be the SVD of…
25
votes
3 answers

How is the null space related to singular value decomposition?

It is said that a matrix's null space can be derived from QR or SVD. I tried an example: $$A= \begin{bmatrix} 1&3\\ 1&2\\ 1&-1\\ 2&1\\ \end{bmatrix} $$ I'm convinced that QR (more precisely, the last two columns of Q) gives the null space: $$Q=…
whitegreen
  • 1,381
  • 1
  • 11
  • 21
24
votes
4 answers

Why does SVD provide the least squares and least norm solution to $ A x = b $?

I am studying the Singular Value Decomposition and its properties. It is widely used in order to solve equations of the form $Ax=b$. I have seen the following: When we have the equation system $Ax=b$, we calculate the SVD of A as $A=U\Sigma V^T$.…
Ufuk Can Bicici
  • 2,706
  • 1
  • 23
  • 45
22
votes
3 answers

How does the SVD solve the least squares problem?

How do I prove that the least-squares solution for $$\text{minimize} \quad \|Ax-b\|_2$$ is $A^{+} b$, where $A^{+}$ is the pseudoinverse of $A$?
22
votes
1 answer

Gradient descent on non-convex function works. How?

For Netflix Prize competition on recommendations one method used a stochastic gradient descent, popularized by Simon Funk who used it to solve an SVD approximately. The math is better explained here on pg 152. A rating is predicted by…
21
votes
4 answers

Strang's proof of SVD and intuition behind matrices $U$ and $V$

In lecture 29 of MIT 18.06, Professor Gilbert Strang "proves" the singular value decomposition (SVD) by assuming that we can write $A = U\Sigma V^T$ and then deriving what $U$, $\Sigma$, and $V$ must be based on the eigendecomposition of $$ AA^T =…
samlaf
  • 800
  • 5
  • 11
20
votes
8 answers

Relation between Cholesky and SVD

When we have a symmetric matrix $A = LL^*$, we can obtain L using Cholesky decomposition of $A$ ($L^*$ is $L$ transposed). Can anyone tell me how we can get this same $L$ using SVD or Eigen decomposition? Thank you.
20
votes
7 answers

Understanding the singular value decomposition (SVD)

Please, would someone be so kind and explain what exactly happens when Singular Value Decomposition is applied on a matrix? What are singular values, left singular, and right singular vectors? I know they are matrices of specific form, I know how to…
Celdor
  • 581
  • 4
  • 14
1
2 3
90 91