The main purpose of this section is a discussion of expected value and covariance for random matrices and vectors. These topics are somewhat specialized, but are particularly important in multivariate statistical models and for the multivariate normal distribution. This section requires some prerequisite knowledge of linear algebra.

We assume that the various indices \( m, \, n, p, k \) that occur in this section are positive integers. Also we assume thatexpected values of real-valued random variables that we reference exist as real numbers, although extensions to cases where expected values are \(\infty\) or \(-\infty\) are straightforward, as long as we avoid the dreaded indeterminate form \(\infty - \infty\).

Basic Theory

Linear Algebra

We will follow our usual convention of denoting random variables by upper case letters and nonrandom variables and constants by lower case letters. In this section, that convention leads to notation that is a bit nonstandard, since the objects that we will be dealing with are vectors and matrices. On the other hand, the notation we will use works well for illustrating the similarities between results for random matrices and the corresponding results in the one-dimensional case. Also, we will try to be careful to explicitly point out the underlying spaces where various objects live.

Let \(\R^{m \times n}\) denote the space of all \(m \times n\) matrices of real numbers. The \( (i, j) \) entry of \( \bs{a} \in \R^{m \times n} \) is denoted \( a_{i j} \) for \( i \in \{1, 2, \ldots, m\} \) and \( j \in \{1, 2, \ldots, n\} \). We will identify \(\R^n\) with \(\R^{n \times 1}\), so that an ordered \(n\)-tuple can also be thought of as an \(n \times 1\) column vector. The transpose of a matrix \(\bs{a} \in \R^{m \times n}\) is denoted \(\bs{a}^T\)—the \( n \times m \) matrix whose \( (i, j) \) entry is the \( (j, i) \) entry of \( \bs{a} \). Recall the definitions of matrix addition, scalar multiplication, and matrix multiplication. Recall also the standard inner product (or dot product) of \( \bs{x}, \, \bs{y} \in \R^n \): \[ \langle \bs{x}, \bs{y} \rangle = \bs{x} \cdot \bs{y} = \bs{x}^T \bs{y} = \sum_{i=1}^n x_i y_i \] The outer product of \( \bs{x} \) and \(\bs{y}\) is \( \bs{x} \bs{y}^T \), the \( n \times n \) matrix whose \( (i, j) \) entry is \( x_i y_j \). Note that the inner product is the trace (sum of the diagonal entries) of the outer product. Finally recall the standard norm on \( \R^n \), given by \[ \|\bs{x}\| = \sqrt{\langle \bs{x}, \bs{x}\rangle} = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2}\] Recall that inner product is bilinear, that is, linear (preserving addition and scalar multiplication) in each argument separately. As a consequence, for \( \bs{x}, \, \bs{y} \in \R^n \), \[ \|\bs{x} + \bs{y}\|^2 = \|\bs{x}\|^2 + \|\bs{y}\|^2 + 2 \langle \bs{x}, \bs{y} \rangle \]

Expected Value of a Random Matrix

As usual, our starting point is a random experiment, modeled by a probability space \((\Omega, \mathscr F, \P)\). So to review, \( \Omega \) is the set of outcomes, \( \mathscr F \) the collection of events, and \( \P \) the probability measure on the sample space \( (\Omega, \mathscr F) \). It's natural to define the expected value of a random matrix in a component-wise manner.

Many of the basic properties of expected value for random variables have analogous results for expected value of random matrices, with matrix operation replacing the ordinary ones. Our first two properties are the critically important linearity properties. The first part is the additive property—the expected value of a sum is the sum of the expected values.

Recall that formula independent, real-valued variables, the expected value of the product is the product of the expected values. Here is the analogous result for random matrices.

Actually the previous result holds if \( \bs{X} \) and \( \bs{Y} \) are simply uncorrelated in the sense that \( X_{i j} \) and \( Y_{j k} \) are uncorrelated for each \( i \in \{1, \ldots, m\} \), \( j \in \{1, 2, \ldots, n\} \) and \( k \in \{1, 2, \ldots p\} \). We will study covariance of random vectors in the next subsection.

Covariance Matrices

Our next goal is to define and study the covariance of two random vectors.

Many of the standard properties of covariance and correlation for real-valued random variables have extensions to random vectors. For the following three results, \( \bs X \) is a random vector in \( \R^m \) and \( \bs Y \) is a random vector in \( \R^n \).

Thus, the covariance of \( \bs{X} \) and \( \bs{Y} \) is the expected value of the outer product of \( \bs{X} - \E(\bs{X}) \) and \( \bs{Y} - \E(\bs{Y}) \). Our next result is the computational formula for covariance: the expected value of the outer product of \( \bs{X} \) and \( \bs{Y} \) minus the outer product of the expected values.

The next result is the matrix version of the symmetry property.

In the following result, \( \bs{0} \) denotes the \( m \times n \) zero matrix.

Naturally, when \( \cov(\bs{X}, \bs{Y}) = \bs{0} \), we say that the random vectors \( \bs{X} \) and \(\bs{Y}\) are uncorrelated. In particular, if the random vectors are independent, then they are uncorrelated. The following results establish the bi-linear properties of covariance.

Variance-Covariance Matrices

The following result is the formula for the variance-covariance matrix of a sum, analogous to the formula for the variance of a sum of real-valued variables.

Recall that \( \var(a X) = a^2 \var(X) \) if \( X \) is a real-valued random variable and \( a \in \R \). Here is the analogous result for the variance-covariance matrix of a random vector.

Recall that if \( X \) is a random variable, then \( \var(X) \ge 0 \), and \( \var(X) = 0 \) if and only if \( X \) is a constant (with probability 1). Here is the analogous result for a random vector:

Recall that since \(\vc(\bs{X})\) is either positive semi-definite or positive definite, the eigenvalues and the determinant of \(\vc(\bs{X})\) are nonnegative. Moreover, if \(\vc(\bs{X})\) is positive semi-definite but not positive definite, then one of the coordinates of \(\bs{X}\) can be written as a linear transformation of the other coordinates (and hence can usually be eliminated in the underlying model). By contrast, if \(\vc(\bs{X})\) is positive definite, then this cannot happen; \(\vc(\bs{X})\) has positive eigenvalues and determinant and is invertible.

Best Linear Predictor

Suppose that \(\bs{X}\) is a random vector in \(\R^m\) and that \(\bs{Y}\) is a random vector in \(\R^n\). We are interested in finding the function of \(\bs{X}\) of the form \(\bs{a} + \bs{b} \bs{X}\), where \(\bs{a} \in \R^n\) and \(\bs{b} \in \R^{n \times m}\), that is closest to \(\bs{Y}\) in the mean square sense. Functions of this form are analogous to linear functions in the single variable case. However, unless \( \bs{a} = \bs{0} \), such functions are not linear transformations in the sense of linear algebra, so the correct term is affine function of \( \bs{X} \). This problem is of fundamental importance in statistics when random vector \(\bs{X}\), the predictor vector is observable, but not random vector \(\bs{Y}\), the response vector. Our discussion here generalizes the one-dimensional case, when \(X\) and \(Y\) are random variables. That problem was solved in the section on covariance. We will assume that \(\vc(\bs{X})\) is positive definite, so that \( \vc(\bs{X}) \) is invertible, and none of the coordinates of \(\bs{X}\) can be written as an affine function of the other coordinates. We write \( \vc^{-1}(\bs{X}) \) for the inverse instead of the clunkier \( \left[\vc(\bs{X})\right]^{-1} \).

As with the single variable case, the solution turns out to be the affine function that has the same expected value as \( \bs{Y} \), and whose covariance with \( \bs{X} \) is the same as that of \( \bs{Y} \).

A simple corollary is the \( \bs{Y} - L(\bs{Y} \mid \bs{X}) \) is uncorrelated with any affine function of \( \bs{X} \):

The variance-covariance matrix of \( L(\bs{Y} \mid \bs{X}) \), and its covariance matrix with \( \bs{Y} \) turn out to be the same, again analogous to the single variable case.

Next is the fundamental result that \( L(\bs{Y} \mid \bs{X}) \) is the affine function of \( \bs{X} \) that is closest to \( \bs{Y} \) in the mean square sense.

The variance-covariance matrix of the difference between \( \bs{Y} \) and the best affine approximation is given in the next theorem.

The actual mean square error when we use \( L(\bs{Y} \mid \bs{X}) \) to approximate \( \bs{Y} \), namely \( \E\left(\left\|\bs{Y} - L(\bs{Y} \mid \bs{X})\right\|^2\right) \), is the trace (sum of the diagonal entries) of the variance-covariance matrix above. The function of \(\bs{x}\) given by \[ L(\bs{Y} \mid \bs{X} = \bs{x}) = \E(\bs{Y}) + \cov(\bs{Y},\bs{X}) \vc^{-1}(\bs{X}) \left[\bs{x} - \E(\bs{X})\right] \] is known as the (distribution) linear regression function. If we observe \(\bs{x}\) then \(L(\bs{Y} \mid \bs{X} = \bs{x})\) is our best affine prediction of \(\bs{Y}\).

Multiple linear regression is more powerful than it may at first appear, because it can be applied to non-linear transformations of the random vectors. That is, if \( g: \R^m \to \R^j \) and \( h: \R^n \to \R^k \) then \( L\left[h(\bs{Y}) \mid g(\bs{X})\right] \) is the affine function of \( g(\bs{X}) \) that is closest to \( h(\bs{Y}) \) in the mean square sense. Of course, we must be able to compute the appropriate means, variances, and covariances.

Moreover, Non-linear regression with a single, real-valued predictor variable can be thought of as a special case of multiple linear regression. Thus, suppose that \(X\) is the predictor variable, \(Y\) is the response variable, and that \((g_1, g_2, \ldots, g_n)\) is a sequence of real-valued functions. We can apply the results of this section to find the linear function of \(\left(g_1(X), g_2(X), \ldots, g_n(X)\right)\) that is closest to \(Y\) in the mean square sense. We just replace \(X_i\) with \(g_i(X)\) for each \(i\). Again, we must be able to compute the appropriate means, variances, and covariances to do this.

Examples and Applications