Multivariate Analysis
Matrix Operations 3

Variance-Maximizing Rotations

  • Consider a bivariate normal distribution with the following contour plot of isodensity ellipses of a bivariate normal distribution:

    Consider

    We wish to find V such that Σy2 is a maximum with the restriction that V'V = I.

    Theorem 1

    Given p variables which follow a p-variate normal distribution, the axis defining the linear transformation with maximum variance is the major axis of the isodensity hyperellipsoid

    Σ is the covariance matrix, and C is an arbitrary positive constant.

    Lagrange Multipliers

    The method of Lagrange multipliers is convient. In this domain, the method is referred to as Eigenvalues and Eigenvectors. This method makes use of the formula,

    in which λ is the eigenvalue and v is the eigenvector. The matrix of v's is used to define the transformation to maximum variance.

    Synonyms

    eigenvalue       characteristic root        latent root
eigenvector      characteristic vector      latent vector

    Solution to Variance-Maximizing Rotations: Eigenvalue Problems

    Consider the equation of the general form, which translates to

    To solve for v, one must first find the value of λ.

    Characteristic Equation

    Eigenvalues found be solving the characteristic equation:

    If A is a 4x4 matrix then the solution to the characteristic equation will involve a 4th degree polynomial and will yield up to four roots.

    The Fundamental Theorem of Algebra

    Every polynomial of degree n, where n is greater than or equal to one, with real or complex coefficients has n real or complex roots.
    Gauss, 1797

    Characteristic Vectors v

    Eigenvectors are found using λ and solving for v.

    Solving for v involves solving a system of p simultaneous equations in p unknowns, such as:

    Rotation for Maximum Variance

    The eignevectors are used to describe the orthogonal rotation from maximum variance. This type of rotation is often referred to as principal-axes rotation.

    Properties of Matrices Related to Eigenvalues and Eigenvectors

    Theorem 2

    The sum Σλi of the eigenvlaues of a matrix A is equal to the trace of A, tr(A).

    Theorem 3

    The product of the λi, of the eigenvlaues of a matrix A is equal to the determinant of A, Det(A).

    Theorem 4

    Two eigenvectors vi and vj associated with two distinct eigenvalues λi and λj of a symmetric matrix are mutually orthogonal, v'ivj = 0.

    Theorem 5

    Given a set of variables X1, X2, ...,Xp, with nonsingular covariance matrix Σ, we can always derive a set of uncorrelated variables Y1, Y2, ..., Yp by a set of linear transforamtions corresponding to the principal-axes rotation. The covariance matrix of this new set of variables is the diagonal matrix Λ = V'ΣV

    Theorem 6

    Given a set of variables X1, X2, ...,Xp, with nonsingular covariance matrix Σx, a new set of variables Y1, Y2, ..., Yp is defined by the transformation Y' = X'V, where V is an orthogonal matrix. If the covariance matrix of the Y's is Σy, then the following relation holds:

    In other words, the quadratic form is invariant under rigid rotation.

    Definition

  • A set of vectors v1, v2, ..., vk is said to be linearly dependent if scalars a1, a2,..., ak, not all zero, can be found such that a1v1 + a2v2 +...+ akvk =0

  • On the other hand, if Σaivi = 0 only when a1 = a2 =...= ak = 0, the set of vectors is said to be linearly independent.

    Another Definition

  • The rank of a pxq matrix is the largest number r such that there exists a set of r rows or columns which is linearly independent.

    Theorem 7

  • (a) The rank of a product C = A*B is less than or equal to the rank of A or B, whichever is smaller.

  • (b) The rank of AA' equals the rank of A'A equals the rank of A.

    Yet Another Definition

    A principal minor of a matrix A is a submatrix obtained by deleting any number of its corresponding row-and-column pairs.

    Example

    has principal minors:

    Definition

  • A matrix G, is said to be Gramian if every principal minor determinant is non-negative.

    Some Properties

  • If G is a nonsingular Gramian matix than all principal minor determinants are positive.

  • If G is a nonsingular Gramian matrix, the G-1 is also a nonsingular Gramian matrix.

    More Properties

  • If G is a Gramian matrix, then for all vectors x, the quadratic form Q = x'Gx has non-negative values, and is said to be positive semidefinite.

  • If G is a nonsingular Gramian matrix then for all vectors x, the quadratic form Q = x'Gx has positive values, and is said to be positive definite.

    Terminology

  • The terms positive semi-definite and positive definite are often applied to the Gramian matrix itself, not just to the quadratic form.

    Yet More Properties

  • All the eigenvlaues of Gramian matrices are non-negative. All the eigenvalues of nonsingular Gramian matrices are positive.

  • Any symmetric matrix whose principal minor determinants are all non-negative is a Gramian matrix (This is the Existence Theorem).

    Theorem 8

    If the matrix A has λ as an eigenvalue and v as the associated eigenvector then the matrix bA has bλ as an eigenvalue, with v as the associated eigenvector.

    Theorem 9

    If the matrix A has λ as an eigenvalue and v as the associated eigenvector then he matrix A + cI has (λ + c) as an eigenvalue, with v as the associated eigenvector.

    Theorem 10

    If the matrix A has λ as an eigenvalue and v as the associated eigenvector then the matrix Am has λm as an eigenvalue, with v as the associated eigenvector.

    Theorem 11

    If the matrix A has &lambda as an eigenvalue and v as the associated eigenvector then the matrix A-1 has 1/λ as an eigenvalue, with v as the associated eigenvector.

    Eigenvalues and Eigenvectors Example

    Consider:

    We are trying to solve (A - λ)v = 0. In order to do this we must have det(A - λI) = 0.

    And

    Setting det(A - λI) = 0 means that

    Thus, the two eigenvalues are λ1 = 4 and λ2 = -1.

    Each eigenvalue has an associated eigenvector, for λ1 = 4 it is found by setting (A - λI)v = 0.

    Both of these equations lead to v2 = (2/3)v1. Thus there are an infinite number of possible solutions that satisify the equation. For simplicity, let's choose v1 = 3 making v2 = 2 and

    For λ2 = -1 the eigenvector is found by setting (A - λI)v = 0.

    These equations imply that v2 = -v1. And again there are an infinite number of solutions. For the sake of simplicity, let v1 = 1 and v2 = -1 giving the eigenvector

    Note: det(A) = 2*1 - 3*2 = 2 - 6 = -4 and λ1*λ2 =4*-1 = -4.

    Note: trace(A) = 2 + 1 = 3 and λ1 + λ2 = 4 + (-1) = 3.

    Try This On Your Own

    Find the eigenvalues and eigenvectors for

    Multivariate Course Page

    Phil Ender, 24apr05, 29Jan98