Slepian's lemma

Mining Engineer (Excluding Oil ) Truman from Alma, loves to spend time knotting, largest property developers in singapore developers in singapore and stamp collecting. Recently had a family visit to Urnes Stave Church. Template:Cleanup

Let $A$ be an m-by-n matrix over a field $\mathbb{𝕂}$, where $\mathbb{𝕂}$, is either the field $\mathbb{ℝ}$, of real numbers or the field $\mathbb{ℂ}$, of complex numbers. Then there is a unique n-by-m matrix $A^{+}$ over $\mathbb{𝕂}$, such that:

1. A A⁺A = A
2. A⁺A A⁺ = A⁺
3. (AA⁺)^* = AA⁺
4. (A⁺A)^* = A⁺A

A⁺ is called the Moore-Penrose pseudoinverse of A. Notice that A is also the Moore-Penrose pseudoinverse of A⁺ . That is, (A⁺ )⁺ = A.

Useful lemmas

In the following lemmas, A is a matrix with complex elements and n columns. B and C are matrices with complex elements and n rows. These results are used in the proofs below.

Lemma 1: A*A = 0 ⇒ A = 0

The assumption says that all elements of A*A are zero. Therefore

$0=\mathrm{Tr}(A^{*}A)=\sum _{j=1}^n(A^{*}A{)}_{jj}=\sum _{j=1}^n\sum _{i=1}^m(A^{*}{)}_{ji}{A}_{ij}=\sum _{i=1}^m\sum _{j=1}^n|A_{ij}{|}^2$.

Therefore all $A_{ij}$ equal 0 i.e. A=0.

Lemma 2: A*AB = 0 ⇒ AB = 0

	0 = A*AB
⇒	0 = B*A*AB
⇒	0 = (AB)*(AB)
⇒	0 = AB   (by Lemma 1)

Lemma 3: BAA* = 0 ⇒ BA = 0

This is proved in a manner similar to the argument of Lemma 2 (or by simply taking the Hermitian conjugate).

Existence and uniqueness

Proof of uniqueness

Suppose that B and C are two n-by-m matrices over $\mathbb{𝕂}$ satisfying the Moore-Penrose criteria. Observe then that

    AB = (AB)* = B*A* = B*(ACA)* = B*A*C*A* = (AB)*(AC)* = ABAC = AC.

Analogously we conclude that BA=CA. The proof is completed by observing that then

    B = BAB = BAC = CAC = C.

Proof of existence

The proof proceeds in stages.

1-by-1 matrices

For any $x\in \mathbb{𝕂}$, we define $x^{+}:=\{\begin{array}{ll}{x}^{-1},& \text{if }x\ne 0\\ 0,& \text{if }x=0\end{array}$

It is easy to see that $x^{+}$ is a pseudoinverse of $x$ (interpreted as a 1-by-1 matrix).

Square diagonal matrices

Let $D$ be an n-by-n matrix over K with zeros off the diagonal. We define $D^{+}$ as an n-by-n matrix over K with $(D^{+}{)}_{ij}:=(D_{ij}{)}^{+}$. We write simply $D_{ij}^{+}$ for $(D^{+}{)}_{ij}=(D_{ij}{)}^{+}$.

Notice that $D^{+}$ is also a matrix with zeros off the diagonal.

We now show that $D^{+}$ is a pseudoinverse of $D$:

1. $(D{D}^{+}D{)}_{ij}=D_{ij}{D}_{ij}^{+}{D}_{ij}=D_{ij}\Rightarrow D{D}^{+}D=D$
2. $(D^{+}D{D}^{+}{)}_{ij}=D_{ij}^{+}{D}_{ij}{D}_{ij}^{+}=D_{ij}^{+}\Rightarrow D^{+}D{D}^{+}=D^{+}$
3. $(D{D}^{+}{)}_{ij}^{*}=\overline{(D{D}^{+}{)}_{ji}}=\overline{D_{ji}{D}_{ji}^{+}}=(D_{ji}{D}_{ji}^{+}{)}^{*}=D_{ji}{D}_{ji}^{+}=D_{ij}{D}_{ij}^{+}\Rightarrow (D{D}^{+}{)}^{*}=D{D}^{+}$
4. $(D^{+}D{)}_{ij}^{*}=\overline{(D^{+}D{)}_{ji}}=\overline{D_{ji}^{+}{D}_{ji}}=(D_{ji}^{+}{D}_{ji}{)}^{*}=D_{ji}^{+}{D}_{ji}=D_{ij}^{+}{D}_{ij}\Rightarrow (D^{+}D{)}^{*}=D^{+}D$

Arbitrary matrices

The singular value decomposition theorem states that there exists a factorization of the form

$A=U\mathrm{\Sigma }{V}^{*}$

where:

U is an m-by-m unitary matrix over K.

Σ is an m-by-n matrix over K with nonnegative numbers on the diagonal and zeros off the diagonal.

V is an n-by-n unitary matrix over K.

Define $A^{+}$ as $V{\mathrm{\Sigma }}^{+}{U}^{*}$.

We now show that $A^{+}$ is a pseudoinverse of $A$:

1. $A{A}^{+}A=U\mathrm{\Sigma }{V}^{*}V{\mathrm{\Sigma }}^{+}{U}^{*}U\mathrm{\Sigma }{V}^{*}=U\mathrm{\Sigma }{\mathrm{\Sigma }}^{+}\mathrm{\Sigma }{V}^{*}=U\mathrm{\Sigma }{V}^{*}=A$
2. $A^{+}A{A}^{+}=V{\mathrm{\Sigma }}^{+}{U}^{*}U\mathrm{\Sigma }{V}^{*}V{\mathrm{\Sigma }}^{+}{U}^{*}=V{\mathrm{\Sigma }}^{+}\mathrm{\Sigma }{\mathrm{\Sigma }}^{+}{U}^{*}=V{\mathrm{\Sigma }}^{+}{U}^{*}=A^{+}$
3. $(A{A}^{+}{)}^{*}=(U\mathrm{\Sigma }{V}^{*}V{\mathrm{\Sigma }}^{+}{U}^{*}{)}^{*}=(U\mathrm{\Sigma }{\mathrm{\Sigma }}^{+}{U}^{*}{)}^{*}=U(\mathrm{\Sigma }{\mathrm{\Sigma }}^{+}{)}^{*}{U}^{*}=U(\mathrm{\Sigma }{\mathrm{\Sigma }}^{+})U^{*}=U\mathrm{\Sigma }{V}^{*}V{\mathrm{\Sigma }}^{+}{U}^{*}=A{A}^{+}$
4. $(A^{+}A{)}^{*}=(V{\mathrm{\Sigma }}^{+}{U}^{*}U\mathrm{\Sigma }{V}^{*}{)}^{*}=(V{\mathrm{\Sigma }}^{+}\mathrm{\Sigma }{V}^{*}{)}^{*}=V({\mathrm{\Sigma }}^{+}\mathrm{\Sigma }{)}^{*}{V}^{*}=V({\mathrm{\Sigma }}^{+}\mathrm{\Sigma })V^{*}=V{\mathrm{\Sigma }}^{+}{U}^{*}U\mathrm{\Sigma }{V}^{*}=A^{+}A$

Basic properties

A*⁺=A^+*

The proof works by showing that A^+* satisfies the four criteria for the pseudoinverse of A*. Since this amounts to just substitution, it is not shown here.

The proof of this relation is given as Exercise 1.18c in.

Identities

A⁺ = A⁺ A^+* A*

A⁺ = A⁺AA⁺ and AA⁺ = (AA⁺)* imply that A⁺ = A⁺(A A⁺)* = A⁺A^+*A*.

A⁺ = A* A^+* A⁺

A⁺ = A⁺AA⁺ and A⁺A = (A⁺A)* imply that A⁺ = (A⁺A)*A⁺ = A*A^+*A⁺.

A = A^+* A* A

A = A A⁺ A and A A⁺ = (A A⁺)* imply that A = (A A⁺)* A = A⁺* A* A.

A = A A* A^+*

A = A A⁺ A and A⁺ A = (A⁺ A)* imply that A = A (A⁺ A)* = A A* A⁺*.

A* = A* A A⁺

This is the conjugate transpose of A = A⁺* A* A above.

A* = A⁺ A A*

This is the conjugate transpose of A = A A* A⁺* above.

Reduction to the Hermitian case

The results of this section show that the computation of the pseudoinverse is reducible to its construction in the Hermitian case. It suffices to show that the putative constructions satisfy the defining criteria.

A⁺ = A* (A A*)⁺

This relation is given as exercise 18(d) in, for the reader to prove, "for every matrix A". Write D = A* (A A*)⁺. Observe that

	AA* = AA*(A A*)+ AA*
⇔	AA* = ADAA*
⇔	0 = (AD − I)AA*
⇔	0 = ADA − A   (by Lemma 3)
⇔	A = ADA.

Similarly, (AA*)⁺AA*(AA*)⁺ = (AA*)⁺ implies that A*(AA*)⁺AA*(AA*)⁺ = A*(AA*)⁺ i.e. DAD = D.

Additionally, AD = AA*(AA*)⁺ so AD = (AD)*.

Finally, DA = A*(AA*)⁺A implies that (DA)* = A* ((AA*)⁺)*A = A* ((AA*)⁺)A = DA.

Therefore D = A⁺.

A⁺ = (A* A)⁺A*

This is proved in an analogous manner to the case above.

Products

In this section, we consider products C = AB.

A has orthonormal columns

If A has orthonormal columns i.e. A*A = I then A⁺=A*. Write D=B⁺A⁺ = B⁺A*. We show that D satisfies the Moore-Penrose criteria.

CDC = ABB⁺A*AB = ABB⁺B = AB = C .

DCD = B⁺A*ABB⁺A* = B⁺BB⁺A* = B⁺A* = D

(CD)* = D*B*A* = A(B⁺)*B*A* = A(BB⁺)*A* = ABB⁺A* = CD

(DC)* = B*A*D* = B*A*A(B⁺)* = (B⁺B)* = B⁺B = B⁺A*AB = DC

Therefore D = C⁺

B has orthonormal rows

If B has orthonormal rows i.e. BB* = I then B⁺=B*. Write D=B⁺A⁺ = B*A⁺. We show that D satisfies the Moore-Penrose criteria.

CDC = ABB*A⁺AB = AA⁺AB = AB = C .

DCD = B*A⁺ABB*A⁺ = B*A⁺AA⁺ = B*A⁺ = D

(CD)* = D*B*A* = (A⁺)*BB*A* = (A⁺)*A* = (AA⁺)* = AA⁺ = ABB*A⁺ = CD

(DC)* = B*A*D* = B*A*(A⁺)*B = B*(A⁺A)*B = B*A⁺AB = DC

Therefore D = C⁺

A has full column rank and B has full row rank

Since A has full column rank, A*A is invertible so (A*A)⁺ = (A*A)⁻¹. Similarly, since B has full row rank, BB* is invertible so (BB*)⁺ = (BB*)⁻¹.

Write D = B⁺A⁺ = B*(BB*)⁻¹(A*A)⁻¹A*. We show that D satisfies the Moore-Penrose criteria.

CDC = ABB*(BB*)⁻¹(A*A)⁻¹A*AB = AB = C .

DCD = B*(BB*)⁻¹(A*A)⁻¹A*ABB*(BB*)⁻¹(A*A)⁻¹A*= B*(BB*)⁻¹(A*A)⁻¹A* = D

CD = ABB*(BB*)⁻¹(A*A)⁻¹)A* = A(AA*)⁻¹A* ⇒ (CD)* = CD.

DC = B*(BB*)⁻¹(A*A)⁻¹)A*AB = B*(BB*)⁻¹B ⇒ (DC)* = DC.

Therefore D = C⁺

Projectors and subspaces

Define P = AA⁺ and Q = A⁺A. Observe that P² = AA⁺AA⁺ = AA⁺ = P. Similarly Q² = Q, and finally, P = P* and Q = Q*. Thus P and Q are orthogonal projection operators. Orthogonality follows from the relations P = P* and Q = Q *. Indeed, consider the operator P: any vector decomposes as

    x = Px + (I-P)x

and for all vectors x and y satisfying Px=x and (I-P)y = y, we have

    x*y = (Px)*(I-P)y = x*P*(I-P)y = x*P(I-P)y = 0.

It follows that PA = AA⁺A = A and A⁺P = A⁺AA⁺ = A⁺. Similarly, QA⁺ = A⁺ and AQ = A. The orthogonal components are now readily identified.

If y belongs to the range of A then for some x, y = Ax and Py = PAx = Ax = y. Conversely, if Py = y then y = AA⁺y so that y belongs to the range of A. It follows that P is the orthogonal projector onto the range of A. I - P is then the orthogonal projector onto the orthogonal complement of the range of A, which equals the kernel of A*.

A similar argument using the relation Q A* = A* establishes that Q is the orthogonal projector onto the range of A* and (I-Q) is the orthogonal projector onto the kernel of A.

Using the relations P(A⁺)* = P*(A⁺)* = (A⁺P)* = (A⁺)* and P = P* = (A⁺)*A* it follows that the range of P equals the range of (A⁺)*, which in turn implies that the range of I-P equals the kernel of A⁺. Similarly QA⁺ = A⁺ implies that the range of Q equals the range of A⁺. Therefore we find,

$\begin{array}{rl}\mathrm{Ker}(A^{+})& =\mathrm{Ker}(A^{*}).\\ \mathrm{\Im }(A^{+})& =\mathrm{\Im }(A^{*}).\\ \end{array}$

Additional properties

Least-squares minimization

In the general case, it is shown here for any $m\times n$ matrix $A$ that $\Vert Ax-b{\Vert }_2\ge \Vert Az-b{\Vert }_2$ where $z=A^{+}b$. This lower bound need not be zero as the system $Ax=b$ may not have a solution (e.g. when the matrix A does not have full rank or the system is overdetermined).

To prove this, we first note that (stating the complex case), using the fact that $P=A{A}^{+}$ satisfies $PA=A$ and $P=P^{*}$, we have

$\begin{array}{rl}{A}^{*}(Az-b)& =A^{*}(A{A}^{+}b-b)\\ & =A^{*}(Pb-b)\\ & =A^{*}{P}^{*}b-A^{*}b\\ & =(PA{)}^{*}b-A^{*}b\\ & =0\end{array}$

so that

$\begin{array}{rl}\Vert Ax-b{\Vert }_2^2& =\Vert Az-b{\Vert }_2^2+(A(x-z){)}^{*}(Az-b)+\text{c.c.}+\Vert A(x-z){\Vert }_2^2\\ & =\Vert Az-b{\Vert }_2^2+(x-z{)}^{*}{A}^{*}(Az-b)+\text{c.c.}+\Vert A(x-z){\Vert }_2^2\\ & =\Vert Az-b{\Vert }_2^2+\Vert A(x-z){\Vert }_2^2\\ & \ge \Vert Az-b{\Vert }_2^2\end{array}$

as claimed.

If $A$ is injective i.e. one-to-one (which implies $m\ge n$), then the bound is attained uniquely at $z$.

Minimum-norm solution to a linear system

The proof above also shows that if the system $Ax=b$ is satisfiable i.e. has a solution, then necessarily $z=A^{+}b$ is a solution (not necessarily unique). We show here that $z$ is the smallest such solution (its Euclidean norm is uniquely minimum).

To see this, note first, with $Q=A^{+}A$, that $Qz=A^{+}A{A}^{+}b=A^{+}b=z$ and that $Q^{*}=Q$. Therefore, assuming that $Ax=b$, we have

$\begin{array}{rl}{z}^{*}(x-z)& =(Qz{)}^{*}(x-z)\\ & =z^{*}Q(x-z)\\ & =z^{*}(A^{+}Ax-z)\\ & =z^{*}(A^{+}b-z)\\ & =0.\end{array}$

Thus

$\begin{array}{rl}\Vert x{\Vert }_2^2& =\Vert z{\Vert }_2^2+2z^{*}(x-z)+\Vert x-z{\Vert }_2^2\\ & =\Vert z{\Vert }_2^2+\Vert x-z{\Vert }_2^2\\ & \ge \Vert z{\Vert }_2^2\end{array}$

with equality if and only if $x=z$, as was to be shown.

References

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.

30 year-old Entertainer or Range Artist Wesley from Drumheller, really loves vehicle, property developers properties for sale in singapore singapore and horse racing. Finds inspiration by traveling to Works of Antoni Gaudí.