Method of Types

Introduction
Let $\mathcal{X}$ be a finite alphabet. Then the type $P_{x^n}$ of a sequence $x^n\in\mathcal{X}^n$ is the empirical frequency of each symbol $a\in\mathcal{X}$ in the sequence $x^n$. That is \begin{equation*} P_{x^n}(a)=\frac{N(a|x^n)}{n}, \end{equation*} where $N(a|x^n)$ is the number of instances of $a$ in the sequence.

Let $P_{n}$ be a type. Then the type class of $P_n$ denoted by $T(P_n)$ is defined as: \begin{equation*} T(P_n)=\{ x^n \in \mathcal{X}^n | P_{x^n} = P_n \}, \end{equation*} where $P_{x^n}$ is the type of $x^n$.

If $X_1, X_2, \cdots, X_n$ are i.i.d. random variables with probability distribution Q(x), then the probability of the sequence $x^n$ under the distribution Q can be easily related to its type: \begin{equation*} Q^n(x^n)=e^{-n( D(P_{x^n}||Q) + H(P_{x^n}) )}, \end{equation*} where $D(\cdotp||\cdotp)$ is the relative entropy or the divergence between two probability distributions and $H(\cdotp)$ is the entropy of the probability distribution $P_{x^n}$. If the distribution Q is in the type class of P_{x^n}, then the above equation reduces to: \begin{equation*} Q^n(x^n)=e^{-nH(P_{x^n})} \end{equation*}

A few Concentration Results
\begin{equation*} \lim_{n \rightarrow \infty} \frac{ \log |T(P_{x^n})| - H(P_{x^n})}{n} = 0. \end{equation*}
 * The size of a type class is equal to $e^{nH(P_{x^n})}$ up to the first order in the exponent:

\begin{equation*} \lim_{n \rightarrow \infty} \frac{ \log |Q^{n}(T(P_{x^n}))| - D(P_{x^n}||Q) } {n} = 0. \end{equation*} This implies that if Q is in the type class of $x^n$, then the type class of $x^n$ has all the probability asymptotically.
 * The probability of the type class under the distribution $Q^n$ is equal to $e^{-nD(P_{x^n}||Q)}$ up to the first order in the exponent:

Applications

 * Rate-Distortion Theory
 * Error Exponents