From Eigenvalues to Singular Values : A Review

The analogy between eigenvalues and singular values has many faces. The current review brings together several examples of this analogy. One example regards the similarity between Symmetric Rayleigh Quotients and Rectangular Rayleigh Quotients. Many useful properties of eigenvalues stem are from the Courant-Fischer minimax theorem, from Weyl’s theorem, and their corollaries. Another aspect regards “rectangular” versions of these theorems. Comparing the properties of Rayleigh Quotient matrices with those of Orthogonal Quotient matrices illuminates the subject in a new light. The Orthogonal Quotients Equality is a recent result that converts Eckart-Young’s minimum norm problem into an equivalent maximum norm problem. This exposes a surprising link between the Eckart-Young theorem and Ky Fan’s maximum principle. We see that the two theorems reflect two sides of the same coin: there exists a more general maximum principle from which both theorems are easily derived. Ky Fan has used his extremum principle (on traces of matrices) to derive analog results on determinants of positive definite Rayleigh Quotients matrices. The new extremum principle extends these results to Rectangular Quotients matrices. Bringing all these topics under one roof provides new insight into the fascinating relations between eigenvalues and singular values.


Introduction
Let G be a real symmetric n n  matrix and let   be a given nonzero vector.Then the well-known Rayleigh Quotient is defined as One motivation behind this definition lies in the following observation.Let  be a given real number.
Then there exists an eigenvalue and the value of  that solves the minimum norm problem (1.3) is given by  .In other words,  provides an estimate for an eigenvalue corresponding to x .Combining this estimate with the inverse iteration yields the celebrated Rayleigh Quotient Iteration.Other related features are the Courant-Fischer minimax inequalities, Weyl monotonicity theorem, and many other results that stem from these observations.In particular, the largest and the smallest eigenvalues of G satisfy respectively.For detailed discussion of the Rayleigh Quotient and its properties see, for example, .
The question that initiates our study is how to extend the definition of Rayleigh Quotient in order to estimate a singular value of a general rectangular matrix, where the term "rectangular" means that the matrix is not necessarily symmetric or square.More precisely, let A   (1.8) so the entries of Z have the same absolute value as the corresponding rectangular quotients.Matrices of the form (1.7) can be viewed as "rectangular" Rayleigh Quotient matrices.The traditional definition of symmetric Rayleigh Quotient matrices refers to symmetric matrices of the form where G and n Y  are defined as above, e.g., [20,30,36].Symmetric Rayleigh Quotient matrices of this form are sometimes called sections.A larger class of Rayleigh Quotients matrices is considered in [34].These matrices have the form where W is a general (nonnormal) square matrix of order n .The n k  matrix C is assumed to be of full column rank, and the k n  matrix I C denotes a left inverse of C .That is, a matrix satisfying I C C I  .
A third class of Rayleigh Quotient matrices is obtained from (1.7) by taking m n k     . These matrices are involved in residual bounds for singular values and singular spaces, e.g., [2,21].
However, our review turns into different directions.It is aimed to explore the optimality properties of Orthogonal Quotients matrices.Comparing these properties with those of symmetric Rayleigh Quotients matrices reveals highly interesting observations.At the heart of these observations stands a surprising relationship between Eckart-Young's minimum norm theorem [5] and Ky Fan's maximum principle [10].
The Eckart-Young theorem considers the problem of approximating one matrix by another matrix of a lower rank.The solution of this problem is also attributed to Schmidt [31].See [17, pp.~137,138] and [33, p.~76].The need for low-rank approximations of a matrix is a fundamental problem that arises in many applications, e.g., [3][4][5]8,14,15,18,33].The maximum principle of Ky Fan considers the problem of maximizing the trace of a symmetric Rayleigh Quotient matrix.It is also a wellknown result that has many applications, e.g., [1,10-12, 16,22,28].Yet, so far, the two theorems have always been considered as independent and unrelated results which are based on different arguments.The Orthogonal Quotients Equality is a recent result that converts Eckart-Young's minimum norm problem into an equivalent maximum norm problem.This exposes a surprising similarity between the Eckart-Young theorem and Ky Fan's maximum principle.We see that the two theorems reflect two sides of the same coin: there exists a more general maximum rule from which both theorems are easily derived.
The plan of our review is as follows.It starts by introducing some necessary notations and facts.Then it turns to expose the basic properties of the Rectangular Quotient (1.6), showing that it solves a number of least norm problems that resemble (1.3).An error bound, similar to (1.2), enables us to bound the distance between  and the closest singular value of A .
Another aspect of the analogy between eigenvalues and singular values is studied in Section 4, in which we consider "rectangular" versions of the Courant-Fischer minimax theorem and Weyl theorem.This paves the way for "traditional" proof of Eckart-Young theorem.Then, using Ky Fan's dominance theorem, it is easy to conclude Mirsky's theorem.
The relation between symmetric Rayleigh Quotient matrices and Orthogonal Quotients matrices is studied in Section 5.It is shown there that the least squares properties of Orthogonal Quotient matrices resemble those of symmetric Rayleigh-Quotient matrices.One consequence of these properties is the Orthogonal Quotients Equality, which is derived in Section 6.As noted above, this equality turns the Eckart-Young least squares problem into an equivalent maximum problem, which attempts to maximize the Frobenius norm of an Orthogonal Quotients matrix of the form (1.7).
The symmetric version of the Orthogonal Quotients Equality considers the problem of maximizing (or minimizing) traces of symmetric Rayleigh Quotients matrices.The solution of these problems is given by the celebrated 10 Ky Fan's Extremum Principles.The similarity between the maximum form of Eckart-Young theorem and Ky Fan's maximum principle suggests that both observations are special cases of a more general extremum principle.The derivation of this principle is carried out in Section 8.It is shown there that both results are easily concluded from the extended maximum principle.
The review ends by discussing some consequences of the extended principle.One consequence is a minimum-maximum equality that relates Mirsky's minimum norm problem with the extended maximum problem.A second type of consequence is about traces of Orthogonal Quotients matrices.The results of Ky Fan on traces of symmetric Rayleigh Quotients matrices [10] were extended in his latter papers [11,12] to products of eigenvalues and determinants.The new extremum principle enables the extension of these properties to Orthogonal Quotients matrices.
The current review brings together several old and new results.The "old" results come with appropriate references.In contrast, the "new" results come without references, as most of them are taken from a recent research paper [3] by this author.Yet the current paper derives a number of contributions which are not included in [3].One contribution regards the extension of Theorem 11 behind the Frobenius norm.Another contribution is the minimum-maximum equality, which is introduced in Section 9.The main difference between [3] and this essay lies in their concept.The first one is a research paper that is aimed at establishing the extended extremum principle.The review exposes some fascinating features of the analogy between eigenvalues and singular values.For this purpose we present several apparently unrelated results.Putting all these topics under one roof gives a better insight into these relations.The description of the results concentrates on real valued matrices and vectors.This simplifies the presentation and helps to focus on the main ideas.The treatment of the complexvalued case should be quite obvious.

Notations and Basic Facts
In this section we introduce notations and facts which are needed for coming discussions.As before A denotes a real m n be an SVD of A , where is an n n  orthogonal matrix, and The singular values of A are assumed to be nonnegative and sorted to satisfy The columns of U and V are called left singular vectors and right singular vectors, respectively.These vectors are related by the equalities T and , 1, , .
A further consequence of (2.1) is the equality Moreover, let r denote the rank of A .Then, clearly, 1 0 and 0 for 1, , .
So (2.4) can be rewritten as be constructed from the first k columns of U and V , respectively.Let a u v , denote the   , i j entries of the matrices , , A U V , respectively.Then (2.4) indicates that 1 for 1, , , .
where the last inequality follows from the Cauchy-Schwarz inequality and the fact that the columns of U and V have unit length.Another useful property regards the concepts of majorization and unitarily invariant norms.Recall that a matrix norm In this case we say that  is weakly majorized by  , or that the singular values of B are weakly majorized by those of C .The Dominance Theorem of Ky Fan [11] relates these two concepts.It says that if the singular values of B are majorized by those of C then the inequality holds for any unitarily invariant norm.For detailed proof of this fact see, for example, [1,11,17,22].The most popular example of a unitarily invariant norm is, perhaps, the Frobenius matrix norm Other examples are the Schatten p -norms, and Ky Fan k -norms, 1 , 1 , , .
The trace norm, tr 1 is obtained for k n  and 1 p  , while the spectral norm (the 2-norm) corresponds to 1 k  and p   .Finally, let m  and n  be a pair of positive integers such that , , , and , , , , and , matrices with orthonormal columns.

Rectangular Quotients
Let A be a real m n  matrix with m n  , and let pair of nonzero vectors.To simplify the coming discussion we make the assumptions that T 0 A  u v , and that u and v are unit vectors.That is, With these assumptions at hand the Rectangular Quotient (1.6) is reduced to the bilinear form In this section we briefly derive the basic minimum norm properties that characterize this kind of bilinear forms.
We shall start by noting that  solves the one parame- This observation is a direct consequence of the equalities Furthermore, substituting the optimal value of  into (3.3)yields the Rectangular Quotient Equality (3.6) which means that solving the rank-one approximation problem is equivalent to solving the maximization problem maximize , subject to 1 and 1.
Using the SVD of A the unit vectors in the last problem can be expressed in the form and , where 1.
Therefore, since T 0 for , the objective function of (3.8) satisfies where the last inequality comes from the Cauchy-Schwarz inequality.Moreover, since T , this pair of vectors solves (3.8), while 1  satisfies The last result is analogous to (1.4).Yet, in contrast to (1.5), here the orthogonality relations (3.9) imply that Further min-max properties of scalar rectangular quotients are obtained from the Courant-Fischer theorem, see the next section.
Another justification behind the proposed definition of the Rectangular Quotient comes from the observation that the Rayleigh quotient corresponding to the matrix Hence in this case the bound (1.2) implies the existence of a singular value of A ,  , that satisfies The last bound can be refined by applying the following retrieval rules, derived in [3].Let n  v  be a given unit vector that satisfies A  v 0 , and let provide the corresponding estimates of a left singular vector, and a singular value, respectively.Then Similarly let m  u  be a given unit vector that satisfies T  A  u 0 , and let denote the corresponding estimates of a singular vector, and a singular value, respectively.Then here We shall finish this section by noting the difference between the Rectangular Quotient (1.6) and the Generalized Rayleigh Quotient (GRQ) proposed by Ostrowski [26].Let W be a general (nonnormal) square matrix of order n and let x and y be two n -vectors that satisfy * 0  x y where * x denotes the conjugate transpose of x .Then the GRQ, is aimed to approximate an eigenvalue of W that is "common" to x and y .For detailed discussions of the GRQ and its properties see [25][26][27]29,40].

From Eigenvalues to Singular Values
The connection between the singular values of A and the eigenvalues of the matrices A T A, AA T , and is quite straightforward.Indeed, many properties of singular values are inherited from this connection.Yet, as our review shows, the depth of the relations is far beyond this basic connection.The Courant-Fischer minimax theorem and Weyl's theorem provide highly useful results on eigenvalues of symmetric matrices.See [1,30] for detailed discussions of these theorems and their consequences.Below we consider the adaption of these results when moving from eigenvalues to singular values.
The adapted theorems are used to provide a "traditional" proof of the Eckart-Young theorem.
As before A denotes a real m n  matrix whose SVD is given by (2.1)-(2.7).The first pair of theorems provides useful "minimax" characterizations of singular values.In these theorems k  denotes an arbitrary subspace of n  that has dimension k .Similarly,   denotes an arbitrary subspace of m  that has dimension  .

Theorem 1 (Right Courant-Fischer Minimax Theorem) The jth singular value of
and where the integer * j is defined by the equality Yet the solutions of both problems are not necessarily unique.

 Theorem 2 (Left Courant-Fischer Minimax Theorem) The ith singular value of
where the integer * i is defined by the equality Moreover, the maximum in (4.4) is attained for while the minimum in (4.5) is attained for


Note that Theorem 2 is essentially Theorem 1 for T A .The proof of Theorem 1 is based on the following idea.The condition (4.3) ensures the existence of a unit vector, x , that belongs both to j  and So the proof is concluded by verifying that equality holds when using the specified subspaces.Let B denote another m n  real matrix and let denote the corresponding difference matrix.The singular values of B and C are denoted as 0 and 0, respectively.The coming corollaries of Theorem 1 answer the question of how the rank of B affects the singular values of where the last inequality follows from (4.2).


Theorem 4 (Weyl) Let B and C be as in (4.7) -(4.8).Then under the convention that 0 while the largest singular value of Let the m n  matrix F be defined by the equality 1 1 .
Hence from (4.11) and (4.12) we see that where the last inequality follows from Lemma 3.

 Corollary 5 (Majorization) Assume further that
Then substituting In other words, if


Recall that k T denotes a rankk Truncated SVD of A , as defined in (2.8).Roughly speaking the last corollary says that a rankk perturbation of A may cause the singular values to "fall" not more than k "levels".The next corollary shows that they are unable to "rise" more than k levels.
Corollary 6 Observe that (4.7) can be rewritten as while B  has the same singular values as B .Hence a further consequence of Theorem 4 is


The next results provide useful bounds on the perturbed singular values.
Corollary 7 (Bounding and Interlacing) Using (4.10) and (4.14) with Furthermore, consider the special case when B is a rank-one matrix.Then using (4.13) and (4.15) with .
Moreover, let k T be a rankk Truncated SVD of A , as defined in (2.8).Then k T solves the minimum norm problem giving the optimal value of .
The last theorem says that k T is a best rankk approximation of A , regarding the Frobenius norm.Observe that Lemma 3 proves a similar claim for the 2norm.The next extension is due to Mirsky [23].
Theorem 9 (Mirsky) Let  denote a unitarily invariant norm on m n   .Then the inequality holds for any matrix In other words, k T solves the minimum norm problem  (4.24) solves this problem.In other words, k R is the smallest perturbation that turns A into a rankk matrix.
Theorem 10 Let B and C be as in (4.7) and assume that Proof.Using the Eckart-Young theorem we obtain Observe that k R remains the solution of (4.23) when this problem is defined by any other unitarily invariant norm.Note also that Total Least Squares problems give rise to a special form of (4.23) in which 1 k r   .In this case the solution matrix, k C  , is reduced to the rank-one matrix T r r r  u v , e.g., [14,15].Further consequences of the Eckart-Young theorem are presented in Section 6.

Least Squares Properties of Orthogonal Quotients Matrices
The optimality properties of symmetric Rayleigh Quotient matrices form the basis of the celebrated Rayleigh-Ritz procedure, e.g., [30,34].In this section we derive the corresponding properties of Orthogonal Quotients matrices.As we shall see, Orthogonal Quotient matrices extend symmetric Rayleigh Quotient matrices in the same way that Rectangular Quotients extend Rayleigh Quotients.
be a given pair of matrices with orthonormal columns, and let denote the corresponding orthogonal quotient matrix.Then Z solves the following three problems.

 
T minimize subject to , where The validity of the last equality is easily verified by noting that the m m  is composed of the first n  rows of the n n  identity matrix.Note also that the corre- The other two problems are solved by similar arguments, using the equalities and where   and , 0 .0


Remark A further inspection of relations (5.7)-(5.9)indicates that Z solves problems (5.3) and (5.4) even if the Frobenius norm is replaced by any other unitarily invariant matrix norm.However the last claim is not valid for problem (5.2), since "punching" a matrix may increase its norm.To see this point consider the matrices 1 1 given pair of matrices with orthonormal columns.Then the matrix

The Orthogonal Quotients Equality and Eckart-Young Theorem
In this section we derive the Orthogonal Quotients .
Proof.Following the proof of Theorem 11 we see that where .

Corollary 16 (The Orthogonal Quotients Equality in Diagonal Form
 be a given pair of matrices with orthonormal columns.Then the diagonal matrix (5.10) satisfies In vector notations the last equality takes the form x y x y x y (6.5)


Let us return now to consider the Eckart-Young problem (4.20).One way to express an m n  matrix, whose rank is at most k , is where minimize , subject to and .
Moreover, both problems are solved by the SVD matrices k U and k V .(See (2.1)-(2.7)for the definition of these matrices.)


The objective functions of the last problems form the left sides of the Orthogonal Quotients Equalities These relations turn the minimum norm problems (6.8)-(6.9)into equivalent maximum norm problems.

Theorem 18 (Maximum Norm Formulations of the Eckart-Young Problem)
The Eckart-Young problems (6.8) and (6.9) are equivalent to the problems 2 T maximize subject to and , respectively.The SVD matrices k U and k V solve both problems.
and the Eckart-Young problem (6.12) can be rewritten as In the next sections we consider extended problems of this type.The key for solving the extended problems lies in the properties of symmetric orthogonal quotients matrices.

The Symmetric Quotients Equality and Ky Fan's Extremum Principles
where  be a given n k  matrix with orthonormal columns and let is invariant under (orthogonal) similarity transformations.
Hence by following the proof of (6.1) we obtain the following results.
The solution of these problems lies in the following well-known properties of symmetric matrices, e.g., [16,30,40].

Theorem 20 (Cauchy Interlace Theorem) Let the k k
 matrix G  be obtained from G by deleting n k    rows and the corresponding  columns.Let , .
In particular, when .
 be constructed from the The minimum trace problem (7.6) is solved by the matrix which is composed of the last k columns of Q .The optimal value of (7.6) is, therefore,


The symmetric quotients equality (7.3) means that Ky Fan's problems, (7.5) and (7.6), are equivalent to the problems respectively.Note the remarkable similarity between Eckart-Young problems (6.8) and (6.12), and Ky Fan problems (7.11) and (7.5), respectively.A further insight is gained by considering the case when G is positive semidefinite.In this case the spectral decomposition (7.1)-(7.2) coincides with the SVD of G and the k k In the next section we move from symmetric orthogonal quotients matrices to rectangular ones.In this case singular values take the role of eigenvalues.Yet, as we shall see, the analogy between the two cases is not always straightforward.

Maximizing (Minimizing) Norms of Orthogonal Quotients Matrices
Let us return to consider orthogonal quotient matrices of the form (5.1).Define and let denote the singular values of the orthogonal quotients matrix subject to and where p is a given positive real number, 0 p    .
Yet the coming results enable us to handle a larger family of objective functions.Perhaps the more interesting problems of this type occur when 1 p  and 2 p  .In these cases the objective function is reduced to respectively.In particular, when 2 p  and m n k     problem (8.1) coincides with the Eckart-Young problem (6.12).The solution of (8.1) and (8.2) is based on "rectangular" extensions of Theorems 20 and 21.The first theorem is due to Thompson [37].We outline its proof to clarify its close relation to Cauchy Interlace Theorem.
Theorem 23 (A Rectangular Cauchy Interlace Theorem) Let the m n    matrix A  be obtained from A by deleting m rows and n columns of and let denote the singular values of A  .Then for 1, , .
Furthermore, the number of positive singular values of A  is bounded from below by  , , and if 1   the first  singular values of A  satisfy the lower bounds Proof.The proof is by induction on   , where is the overall number of deleted rows and columns.For


Observe that the bounds (8.5) and (8.6) are "strict" in the sense that these bounds can be satisfied as equalities.Take, for example, a diagonal matrix.

Corollary 24 (A Rectangular Poincaré Separation Theorem) Consider the m n
the first  singular values of A  satisfy the lower bounds .
1, , , and Similar inequalities hold when the power function be constructed from the first m  columns of U , and let the n n  be constructed from the first n  columns of V .(Recall that U and V form the SVD of A , see (2.1)-(2.8).)Then this pair of matrices solves the maximum problem (8.1), giving the optimal value of However, the solution matrices are not necessarily unique.
Proof.The proof is a direct consequence of (8.10) and the fact that T

 Corollary 27 (A rectangular Ky Fan Maximum
Principle) Consider the special case when 1 p  .In this case and the optimal value is attained for the matrices m U  and n V  .Corollary 28 Consider the special case when and the optimal value is attained for the matrices m U  and n V  .Furthermore, if m n k     then (8.15) is reduced to (6.12).This gives an alternative way to prove the Eckart Observe that the remaining nonzero entries of T M SN are the singular values of this matrix.Note also that the rule for deleting rows and columns from S is aimed to make the size of the remaining nonzero entries as small as possible: The product


Another pair of matrices that solve (8.2) is gained by reversing the order in which we delete rows and columns from S : Start by deleting the first n columns of S , which contain the n largest singular values.Then dele- te the m rows of S that contain the next m largest singular values.
Corollary 30 (A rectangular Ky Fan Minimum Principle) Consider the special case when 1 p  and 1   .In this case and the optimal value is attained for the matrices ˆm U  and ˆn V  .Corollary 31 Consider the special case when and the optimal value is attained for the matrices ˆm U  and ˆn V  .The next theorem extends our results to arbitrary unitarily invariant norms.
Proof.The optimal value of the minimized term is given by Theorem 9 (Mirsky's theorem), and this value equals We have seen that Mirsky's minimization problem (4.22) and the maximum problem (8.18) share a common feature: The optimal values of both problems are obtained for the SVD matrices, k U and k V .This observation enables us to extend the minimum-maximum equality to other unitarily invariant norms.Consider, for example, the spectral norm (2.19).In this case (9.3) is replaced with Recall further that the jth columns of the SVD matrices form a pair of singular vectors that correspond to j  .Indeed, it is this property that ensures the equality in (9.2).This paves the way for another variant of the Orthogonal Quotients Equality.
Theorem 34 Let the matrices  be composed from pairs of singular vectors of A .That is,

Traces of Rectangular Matrices
We have seen that Ky Fan's extremum principles maximize and minimize traces of symmetric Rayleigh Quotient matrices.In this section we bring analog results in terms of rectangular matrices.For this purpose we define the trace of a rectangular m n  matrix as   trace .giving the optimal value of Finally we note that when m n k     the matrix T m n X AY   turns to be a square matrix and Corollary 35 is reduced to the following known result., and , respectively.Hence from (11.2) we see that where k U and k V are defined in (2.7).


The solution of (11.9) is found by following the notations and the proof of Theorem 29.

The number of positive singular values of the matrix
Otherwise, when k   ,

Concluding Remarks
According to an old adage, the whole can sometimes be much more than the sum of its parts.The Rayleigh-Ritz procedure, the Eckart-Young theorem, and Ky Fan maximum principle are fundamental results that have several applications.The observation that these topics are closely related is new and surprising.It illuminates these issues in a new light.The extended maximum principle is a powerful tool that has important consequences.In particular we see that both Eckart-Young's maximum problem and Ky Fan's maximum problem are special cases of this observation.The minimum-maximum theorem connects the extended maximum problem with Mirsky's minimum norm problem.
The review provides a second look at results of Ky Fan that consider eigenvalues of symmetric Rayleigh Quotient matrices.It extends these results to "rectangular" versions that consider singular values of Orthogonal Quotients matrices.The proofs illustrate the usefulness of Ky Fan's dominance theorem.With this theorem at hand Mirsky's theorem is easily derived from Weyl theorem.Similarly, it helps to establish the extended extremum principle.

11 C
be a given pair of m n  matrices with singular val-

(
The maximum in (4.1) is over all j dimensional subspaces j  of n  .The minimum in (4.2) is over all * j dimensional subspaces * j  of n  .)Moreover, the maximum in (4.1) is attained for the singular values of C majorize the singular values of k A T  .

Theorem 8 (
Eckart-Young) Let B and C be as in (4.7)-(4.8)and assume that

22 )
Proof.From Corollary 5 we see that the singular values of A B  majorize those of k A T  .Hence (4.21) is a direct consequence of Ky Fan Dominance Theorem. Another related problem is denotes the set of all real m n  matrices of rank k .Below we will show that the residual matrix T 1

. 4 )
Proof.Completing the columns of m X  to be an orthonormal basis of m  gives an orthogonal m m  matrix, m X , whose first m  columns are the columns of m X  .Similarly there exists an orthogonal n n  ma- trix, n Y , whose first n  columns are the columns of n Y  .Therefore, since the Frobenius norm is unitarily invariant,

whose trace norms are 2 and 5 ,
respectively.That is, punching A increases its trace norm.Nevertheless, punching a matrix always reduces its Frobenius norm.Hence the proof of Theorem 11 leads to the following powerful results.Corollary 12 Let m n   denote the set of all real m n    matrices that have a certain pattern of zeros.(For example, the set of all tridiagonal matrices.)Let the matrix


As in the scalar case, the residual functions, to bound the "distance" between the singular values of R and A .Assume for a moment that m values of R .Then there exists a permutation  of 14) see[2,21].The relations (5.7)-(5.9)indicate that the minimal values of     and   R  equal the Frobenius norm of the corresponding off-diagonal blocks in the matrix T m n X AY .The next observations regard the minimal value of   R  .
Equality and discuss its relation to the Eckart-Young theorem. k

Theorem 19 (
The Symmetric Quotients Equality) Using the above notations, Note that Theorem 19 remains valid when G is replaced by any real n n  matrix.The role of symmetry becomes prominent in problems that attempt to maximize or minimize , as considered by Ky Fan [10].In our notations Ky Fan's problems have the form of Q .Then k Q solves (7.5), giving the optimal value of

. 13 )
denote the eigenvalues (the singular values) of this matrix.Then here the interlacing relations (7.7) imply majorization relations between the singular values of T k k Y GY and the singular values of the matrices T cases to consider.Assume first that A  is obtained by deleting one row of A .Then using Theorem 20 with T G AA  and T G AA     gives the desired results.The second possibility is that A  is obtained by deleting one column of A .In this case Theorem 20 is used with T .Similar arguments enable us to complete the induction step.

)
Proof.Let the matrix m m X   be obtained by completing the columns of m X  to be an orthonormal basis of m  .Let the matrix n n Y   be obtained by completing the columns of n Y  to be an orthonormal basis of n  .Then the m n  matrix T m n X AY has the same singular values as A , and A  is obtained from T m nX AY be deleting the last m rows and the last n columns.Corollary 25 Using the former notations,

TMS
deletes the first m rows of S , which contain the largest m singular values.Then the product   T M S N annihilates the next n largest singular values.The remaining nonzero entries of S are, therefore, the smallest that we can get.The number of positive singular values in T  .The optimality of our solution stems from (8.8) and (8.12).


. The optimal value of the other problem is determined by the maximum principle (8.13), and this value power function (9.1) coincides with the trace norm(2.18).In this case (9.2) yields the following elegant result.
this definition at hand the new problems to solve are

10 . 4 )
On the other hand, the matrices m U  and n V the following conclusions.Corollary35 The matrices m U  and n V  solve (10.2)

10 . 7 ) 36
CorollaryThe matrices m U   and n V  (or m U  and n V   ) solve (10.3) giving the optimal value of

2 )
Let the matrices k Q and ˆk Q be defined as in Corollary 22. Then the eigenvalues of the matrices T

7 )(
are attained for the matrices k Q and ˆk Q , respectively.A further strengthening of (11.4) is gained by applying Hadamard determinant theorem, which says that the determinant of a symmetric positive semidefinite matrix, T k k Y GY , is smaller than the product of its diagonal entries.That is, and the optimal value is attained for ˆk Q .Let us return now to consider rectangular orthogonal quotients matrices.Using the notations of Section 8, the problems that we want to solve are Compare with (8.1) and (8.2), respectively.)Let the matrices m U  and n V  be defined as in Theorem 26.Using (2.1) one can verify that maximal possible value, see(8.7).This brings us to the following conclusions.
sible only when r = n.If m n  then the equalities k   and r = n imply 0 m  and m m Alternatively we can write B in the form