Random samples from normal distributions are the most important special cases of the topics in this chapter. As we will see, many of the results simplify significantly when the underlying sampling distribution is normal. In addition we will derive the distributions of a number of random variables constructed from normal samples that are of fundamental important in inferential statistics.

The One Sample Model

Suppose that $\mathit{X}=(X_1,X_2,\dots ,X_n)$ is a random sample from the normal distribution with mean $\mu \in \mathbb{R}$ and standard deviation $\sigma \in (0,\mathrm{\infty })$. Recall that the term random sample means that $\mathit{X}$ is a sequence of independent, identically distributed random variables. Recall also that the normal distribution has probability density function $$f(x)=\frac{1}{\sqrt{2\pi }\sigma }\exp [-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2],x\in \mathbb{R}$$ In the notation that we have used elsewhere in this chapter, ${\sigma }_3=\mathbb{E}[(X-\mu {)}^3]=0$ (equivalently, the skewness of the normal distribution is 0) and ${\sigma }_4=\mathbb{E}[(X-\mu {)}^4]=3{\sigma }^4$ (equivalently, the kurtosis of the normal distribution is 3). Since the sample (and in particular the sample size $n$) is fixed is this subsection, it will be suppressed in the notation.

The Sample Mean

First recall that the sample mean is $$M=\frac{1}{n}\sum _{i=1}^n{X}_i$$

Of course, by the central limit theorem, the distribution of $M$ is approximately normal, if $n$ is large, even if the underlying sampling distribution is not normal.

The standard score $Z$ in [2] plays a critical role in constructing interval estimates and hypothesis tests for the distribution mean $\mu$ when the distribution standard deviation $\sigma$ is known. The random variable $Z$ will also appear in several derivations in this section.

The Sample Variance

The main goal of this subsection is to show that certain multiples of the two versions of the sample variance that we have studied have chi-square distributions. Recall that the chi-square distribution with $k\in {\mathbb{N}}_{+}$ degrees of freedom has probability density function $$f(x)=\frac{1}{\mathrm{\Gamma }(k/2)2^{k/2}}{x}^{k/2-1}{e}^{-x/2},0<x<\mathrm{\infty }$$ and has mean $k$ and variance $2k$. The moment generating function is $$G(t)=\frac{1}{(1-2t{)}^{k/2}},-\mathrm{\infty }<t<\frac{1}{2}$$ The most important result to remember is that the chi-square distribution with $k$ degrees of freedom governs $\sum _{i=1}^k{Z}_i^2$, where $(Z_1,Z_2,\dots ,Z_k)$ is a sequence of independent, standard normal random variables.

Recall that if $\mu$ is known, a natural estimator of the variance ${\sigma }^2$ is the statistic $$W^2=\frac{1}{n}\sum _{i=1}^n(X_i-\mu {)}^2$$ Although the assumption that $\mu$ is known is almost always artificial, $W^2$ is very easy to analyze and it will be used in some of the derivations below. Our first result is the distribution of a simple multiple of $W^2$.

The variable $U$ in [3] plays a critical role in constructing interval estimates and hypothesis tests for the distribution standard deviation $\sigma$ when the distribution mean $\mu$ is known (although again, this assumption is usually not realistic).

As an estimator of ${\sigma }^2$, part (a) means that $W^2$ is unbiased and part (b) means that $W^2$ is consistent. Of course, these moment results are special cases of the general results obtained in the section on Sample Variance. In that section, we also showed that $M$ and $W^2$ are uncorrelated if the underlying sampling distribution has skewness 0 (${\sigma }_3=0$), as is the case here.

Recall now that the standard version of the sample variance is the statistic $$S^2=\frac{1}{n-1}\sum _{i=1}^n(X_i-M{)}^2$$ The sample variance $S^2$ is the usual estimator of ${\sigma }^2$ when $\mu$ is unknown (which is usually the case). We showed earlier that in general, the sample mean $M$ and the sample variance $S^2$ are uncorrelated if the underlying sampling distribution has skewness 0 (${\sigma }_3=0$). It turns out that if the sampling distribution is normal, these variables are in fact independent, a very important and useful property, and at first blush, a very surprising result since $S^2$ appears to depend explicitly on $M$.

We can now determine the distribution of a simple multiple of the sample variance $S^2$.

The variable $V$ in [6] plays a critical role in constructing interval estimates and hypothesis tests for the distribution standard deviation $\sigma$ when the distribution mean $\mu$ is unknown (almost always the case).

As before, these moment results are special cases of the general results obtained in the section on Sample Variance. Again, as an estimator of ${\sigma }^2$, part (a) means that $S^2$ is unbiased, and part (b) means that $S^2$ is consistent. Note also that $\text{var}(S^2)$ is larger than $\text{var}(W^2)$ (not surprising), by a factor of $\frac{n}{n-1}$.

Note that the correlation does not depend on the parameters $\mu$ and $\sigma$, and converges to 1 as $n\to \mathrm{\infty }$,

The $T$ Variable

Recall that the Student $t$ distribution with $k\in {\mathbb{N}}_{+}$ degrees of freedom has probability density function $$f(t)=C_k{(1+\frac{t^2}{k})}^{-(k+1)/2},t\in \mathbb{R}$$ where $C_k$ is the appropriate normalizing constant. The distribution has mean 0 if $k>1$ and variance $k/(k-2)$ if $k>2$. In this subsection, the main point to remember is that the $t$ distribution with $k$ degrees of freedom is the distribution of $$\frac{Z}{\sqrt{V/k}}$$ where $Z$ has the standard normal distribution; $V$ has the chi-square distribution with $k$ degrees of freedom; and $Z$ and $V$ are independent.

The Two Sample Model

Suppose that $\mathit{X}=(X_1,X_2,\dots ,X_m)$ is a random sample of size $m$ from the normal distribution with mean $\mu \in \mathbb{R}$ and standard deviation $\sigma \in (0,\mathrm{\infty })$, and that $\mathit{Y}=(Y_1,Y_2,\dots ,Y_n)$ is a random sample of size $n$ from the normal distribution with mean $\nu \in \mathbb{R}$ and standard deviation $\tau \in (0,\mathrm{\infty })$. Finally, suppose that $\mathit{X}$ and $\mathit{Y}$ are independent. Of course, all of the results above in the one sample model in apply to $\mathit{X}$ and $\mathit{Y}$ separately, but now we are interested in statistics that are helpful in inferential procedures that compare the two normal distributions. We will use the basic notation established above, but we will indicate the dependence on the sample.

The two-sample (or more generally the multi-sample model) occurs naturally when a basic variable in the statistical experiment is filtered according to one or more other variable (often nominal variables). For example, in the cicada data, the weights of the male cicadas and the weights of the female cicadas may fit observations from the two-sample normal model. The basic variable weight is filtered by the variable gender. If weight is filtered by gender and species, we might have observations from the 6-sample normal model.

The Difference in the Sample Means

We know from [1] that $M(\mathit{X})$ and $M(\mathit{Y})$ have normal distributions. Moreover, these sample means are independent because the underlying samples $\mathit{X}$ and $\mathit{Y}$ are independent. Hence, it follows from a basic property of the normal distribution that any linear combination of $M(\mathit{X})$ and $M(\mathit{Y})$ will be normally distributed as well. For inferential procedures that compare the distribution means $\mu$ and $\nu$, the linear combination that is most important is the difference.

Hence the standard score $$Z=\frac{[(M(\mathit{X})-M(\mathit{Y})]-(\mu -\nu )}{\sqrt{{\sigma }^2/m+{\tau }^2/n}}$$ has the standard normal distribution. This standard score plays a fundamental role in constructing interval estimates and hypothesis test for the difference $\mu -\nu$ when the distribution standard deviations $\sigma$ and $\tau$ are known.

Ratios of Sample Variances

Next we will show that the ratios of certain multiples of the sample variances (both versions) of $\mathit{X}$ and $\mathit{Y}$ have $F$ distributions. Recall that the $F$ distribution with $j\in {\mathbb{N}}_{+}$ degrees of freedom in the numerator and $k\in {\mathbb{N}}_{+}$ degrees of freedom in the denominator is the distribution of $$\frac{U/j}{V/k}$$ where $U$ has the chi-square distribution with $j$ degrees of freedom; $V$ has the chi-square distribution with $k$ degrees of freedom; and $U$ and $V$ are independent. The $F$ distribution is named in honor of Ronald Fisher and has probability density function $$f(x)=C_{j,k}\frac{x^{(j-2)/2}}{{[1+(j/k)x]}^{(j+k)/2}},0<x<\mathrm{\infty }$$ where $C_{j,k}$ is the appropriate normalizing constant. The mean is $\frac{k}{k-2}$ if $k>2$, and the variance is $2{\left(\frac{k}{k-2}\right)}^2\frac{j+k-2}{j(k-4)}$ if $k>4$.

These variables are useful for constructing interval estimates and hypothesis tests of the ratio of the standard deviations $\sigma /\tau$. The choice of the $F$ variable depends on whether the means $\mu$ and $\nu$ are known or unknown. Usually, of course, the means are unknown and so the statistic in in [15] is used.

The $T$ Variable

Our final construction in the two sample normal model will result in a variable that has the student $t$ distribution. This variable plays a fundamental role in constructing interval estimates and hypothesis test for the difference $\mu -\nu$ when the distribution standard deviations $\sigma$ and $\tau$ are unknown. The construction requires the additional assumption that the distribution standard deviations are the same: $\sigma =\tau$. This assumption is reasonable if there is an inherent variability in the measurement variables that does not change even when different treatments are applied to the objects in the population.

To construct our desired variable, we first need an estimate of ${\sigma }^2$. A natural approach is to consider a weighted average of the sample variances $S^2(\mathit{X})$ and $S^2(\mathit{Y})$, with the degrees of freedom as the weight factors.

The Bivariate Sample Model

Suppose now that $((X_1,Y_1),(X_2,Y_2),\dots ,(X_n,Y_n))$ is a random sample of size $n$ from the bivariate normal distribution with means $\mu \in \mathbb{R}$ and $\nu \in \mathbb{R}$, standard deviations $\sigma \in (0,\mathrm{\infty })$ and $\tau \in (0,\mathrm{\infty })$, and correlation $\rho \in [0,1]$. Of course, $\mathit{X}=(X_1,X_2,\dots ,X_n)$ is a random sample of size $n$ from the normal distribution with mean $\mu$ and standard deviation $\sigma$, and $\mathit{Y}=(Y_1,Y_2,\dots ,Y_n)$ is a random sample of size $n$ from the normal distribution with mean $\nu$ and standard deviation $\tau$, so the results above in the one sample model in apply to $\mathit{X}$ and $\mathit{Y}$ individually. Thus our interest in this section is in the relation between various $\mathit{X}$ and $\mathit{Y}$ statistics and properties of sample covariance.

The bivariate (or more generally multivariate) model occurs naturally when considering two (or more) variables in the statistical experiment. For example, the heights of the fathers and the heights of the sons in Pearson's height data may well fit observations from the bivariate normal model.

In the notation that we have used previously, recall that ${\sigma }^3=\mathbb{E}[(X-\mu {)}^3]=0$, ${\sigma }_4=\mathbb{E}[(X-\mu {)}^4]=3{\sigma }^4$, ${\tau }_3=\mathbb{E}[(Y-\nu {)}^3]=0$, ${\tau }_4=\mathbb{E}[(Y-\nu {)}^4]=3{\tau }^4$, $\delta =\text{cov}(X,Y)=\sigma \tau \rho$. and ${\delta }_2=\mathbb{E}[(X-\mu {)}^2(Y-\nu {)}^2]={\sigma }^2{\tau }^2(1+2{\rho }^2)$.

Sample Means

Of course, we know the individual means and variances of $M(\mathit{X})$ and $M(\mathit{Y})$ from the one-sample model above. Hence we know the complete distribution of $(M(\mathit{X}),M(\mathit{Y}))$.

Sample Variances

Sample Covariance

If $\mu$ and $\nu$ are known (again usually an artificial assumption), a natural estimator of the distribution covariance $\delta$ is the special version of the sample covariance $$W(\mathit{X},\mathit{Y})=\frac{1}{n}\sum _{i=1}^n(X_i-\mu )(Y_i-\nu )$$

If $\mu$ and $\nu$ are unknown (again usually the case), then a natural estimator of the distribution covariance $\delta$ is the standard sample covariance $$S(\mathit{X},\mathit{Y})=\frac{1}{n-1}\sum _{i=1}^n[X_i-M(\mathit{X})][Y_i-M(\mathit{Y})]$$

Computational Exercises

We use the basic notation established above for samples $\mathit{X}$ and $\mathit{Y}$, and for the statistics $M$, $W^2$, $S^2$, $T$ , and so forth.