The Multinomial Distribution

A multinomial trials process is a sequence of independent, identically distributed random variables $X = (X_{1}, X_{2}, \dots)$ each wtih $k$ possible values. For simplicity, we will denote the set of outcomes by ${1, 2, \dots, k}$ , and we will denote the common probability density function of the trial variables by $p_{i} = P (X_{j} = i), i \in {1, 2, \dots, k}$

Thus, the multinomial trials process is a simple generalization of the Bernoulli trials process (which corresponds to

k = 2

). Of course

p_{i} > 0

for each

i

and

\sum_{i = 1}^{k} p_{i} = 1

. In statistical terms, the sequence

X

is formed by sampling from the distribution.

As with our discussion of the binomial distribution, we are interested in the random variables that count the number of times each outcome occurred.

Let $Y_{i} = # {j \in {1, 2, \dots, n} : X_{j} = i} = \sum_{j = 1}^{n} 1 (X_{j} = i), i \in {1, 2, \dots, k}$

Of course, these random variables also depend on the parameter

n

(the number of trials), but this parameter is fixed in our discussion so we suppress it to keep the notation simple. Note that

\sum_{i = 1}^{k} Y_{i} = n

so if we know the values of

k - 1

of the counting variables, we can find the value of the remaining variable.

Basic arguments using independence and combinatorics can be used to derive the joint, marginal, and conditional densities of the counting variables. In particular, recall the definition of the multinomial coefficient: for nonnegative integers

(j_{1}, j_{2}, \dots, j_{n})

with

\sum_{i = 1}^{k} j_{i} = n

(\binom{n}{j_{1}, j_{2}, \dots, j_{k}}) = \frac{n!}{j_{1}! j_{2}! \dots j_{k}!}

Distributions

For nonnegative integers $(j_{1}, j_{2}, \dots, j_{k})$ with $\sum_{i = 1}^{k} j_{i} = n$ , $P (Y_{1} = j_{1}, Y_{2}, = j_{2} \dots, Y_{k} = j_{k}) = (\binom{n}{j_{1}, j_{2}, \dots, j_{k}}) p_{1}^{j_{1}} p_{2}^{j_{2}} \dots p_{k}^{j_{k}}$

Details:

By independence, any sequence of trials in which outcome $i$ occurs exactly $j_{i}$ times for $i \in {1, 2, \dots, k}$ has probability $p_{1}^{j_{1}} p_{2}^{j_{2}} \dots p_{k}^{j_{k}}$ . The number of such sequences is the multinomial coefficient $(\binom{n}{j_{1}, j_{2}, \dots, j_{k}})$ . Thus, the result follows from the additive property of probability.

The distribution of

Y = (Y_{1}, Y_{2}, \dots, Y_{k})

is called the multinomial distribution with parameters

n

and

p = (p_{1}, p_{2}, \dots, p_{k})

. We also say that

(Y_{1}, Y_{2}, \dots, Y_{k - 1})

has this distribution (recall that the values of

k - 1

of the counting variables determine the value of the remaining variable). Usually, it is clear from context which meaning of the term multinomial distribution is intended. Again, the ordinary binomial distribution corresponds to

k = 2

For each $i \in {1, 2, \dots, k}$ , $Y_{i}$ has the binomial distribution with parameters $n$ and $p_{i}$ : $P (Y_{i} = j) = (\binom{n}{j}) p_{i}^{j} (1 - p_{i})^{n - j}, j \in {0, 1, \dots, n}$

Details:

There is a simple probabilistic proof. If we think of each trial as resulting in outcome $i$ or not, then clearly we have a sequence of $n$ Bernoulli trials with success parameter $p_{i}$ . Random variable $Y_{i}$ is the number of successes in the $n$ trials. The result could also be obtained by summing the joint probability density function in [3] 1 over all of the other variables, but this would be much harder.

The multinomial distribution is preserved when the counting variables are combined.

Suppose that $(A_{1}, A_{2}, \dots, A_{m})$ is a partition of the index set ${1, 2, \dots, k}$ into nonempty subsets. For $j \in {1, 2, \dots, m}$ let $Z_{j} = \sum_{i \in A_{j}} Y_{i}, q_{j} = \sum_{i \in A_{j}} p_{i}$ Then $Z = (Z_{1}, Z_{2}, \dots, Z_{m})$ has the multinomial distribution with parameters $n$ and $q = (q_{1}, q_{2}, \dots, q_{m})$ .

Details:

Again, there is a simple probabilistic proof. Each trial, independently of the others, results in an outome in $A_{j}$ with probability $q_{j}$ . For each $j$ , $Z_{j}$ counts the number of trails which result in an outcome in $A_{j}$ . This result could also be derived from the joint probability density function in [3] , but again, this would be a much harder proof.

The multinomial distribution is also preserved when some of the counting variables are observed.

Suppose that $(A, B)$ is a partition of the index set ${1, 2, \dots, k}$ into nonempty subsets. Suppose that $(j_{i} : i \in B)$ is a sequence of nonnegative integers, indexed by $B$ such that $j = \sum_{i \in B} j_{i} \leq n$ . Let $p = \sum_{i \in A} p_{i}$ . Then the conditional distribution of $(Y_{i} : i \in A)$ given $(Y_{i} = j_{i} : i \in B)$ is multinomial with parameters $n - j$ and $(p_{i} / p : i \in A)$ .

Details:

Again, there is a simple probabilistic argument and a harder analytic argument. If we know $Y_{i} = j_{i}$ for $i \in B$ , then there are $n - j$ trials remaining, each of which, independently of the others, must result in an outcome in $A$ . The conditional probability of a trial resulting in $i \in A$ is $p_{i} / p$ .

Combinations of the basic results involving grouping in [5] and conditioning in [6] can be used to compute any marginal or conditional distributions.

Moments

We will compute the mean and variance of each counting variable, and the covariance and correlation of each pair of variables.

For $i \in {1, 2, \dots, k}$ , the mean and variance of $Y_{i}$ are

$E (Y_{i}) = n p_{i}$
$var (Y_{i}) = n p_{i} (1 - p_{i})$

Details:

Recall that $Y_{i}$ has the binomial distribution with parameters $n$ and $p_{i}$ .

For distinct $i, j \in {1, 2, \dots, k}$ ,

$cov (Y_{i}, Y_{j}) = - n p_{i} p_{j}$
$cor (Y_{i}, Y_{j}) = - \sqrt{p_{i} p_{j} / [(1 - p_{i}) (1 - p_{j})]}$

Details:

From the bi-linearity of the covariance operator, we have $cov (Y_{i}, Y_{j}) = \sum_{s = 1}^{n} \sum_{t = 1}^{n} cov [1 (X_{s} = i), 1 (X_{t} = j)]$ If $s = t$ , the covariance of the indicator variables is $- p_{i} p_{j}$ . If $s \neq t$ the covariance is 0 by independence. Part (b) can be obtained from part (a) using the definition of correlation and the variances of $Y_{i}$ and $Y_{j}$ given in [7].

From [8], note that the number of times outcome

i

occurs and the number of times outcome

j

occurs are negatively correlated, but the correlation does not depend on

n

If $k = 2$ , then the number of times outcome 1 occurs and the number of times outcome 2 occurs are perfectly correlated.

Details:

This follows immediately from [8] since we must have $i = 1$ and $j = 2$ , and $p_{2} = 1 - p_{1}$ . Of course we can also argue this directly since $Y_{2} = n - Y_{1}$ .

Examples and Applications

In the dice experiment, select the number of aces. For each die distribution, start with a single die and add dice one at a time, noting the shape of the probability density function and the size and location of the mean/standard deviation bar. When you get to 10 dice, run the simulation 1000 times and compare the relative frequency function to the probability density function, and the empirical moments to the distribution moments.

Suppose that we throw 10 standard, fair dice. Find the probability of each of the following events:

Scores 1 and 6 occur once each and the other scores occur twice each.
Scores 2 and 4 occur 3 times each.
There are 4 even scores and 6 odd scores.
Scores 1 and 3 occur twice each given that score 2 occurs once and score 5 three times.

Details:

0.00375
0.0178
0.205
0.0879

Suppose that we roll 4 ace-six flat dice (faces 1 and 6 have probability $\frac{1}{4}$ each; faces 2, 3, 4, and 5 have probability $\frac{1}{8}$ each). Find the joint probability density function of the number of times each score occurs.

Details:

$f (u, v, w, x, y, z) = (\binom{4}{u, v, w, x, y, z}) {(\frac{1}{4})}^{u + z} {(\frac{1}{8})}^{v + w + x + y}$ for nonnegative integers $u, v, w, x, y, z$ that sum to 4

In the dice experiment, select 4 ace-six flats. Run the experiment 500 times and compute the joint relative frequency function of the number times each score occurs. Compare the relative frequency function to the true probability density function.

Suppose that we roll 20 ace-six flat dice. Find the covariance and correlation of the number of 1's and the number of 2's.

Details:

covariance: $- 0.625$ ; correlation: $- 0.0386$

In the dice experiment, select 20 ace-six flat dice. Run the experiment 500 times, updating after each run. Compute the empirical covariance and correlation of the number of 1's and the number of 2's. Compare the results with the theoretical results in [14].

5. The Multinomial Distribution

Basic Theory

Multinomial trials

Distributions

Moments

Examples and Applications