Typicality

Typicality is a term used in information theory to refer to measures of how close the empirical distribution of a sequence is to a given distribution.

= Flavors of typicality =

Let $x^n= (x_1, x_2, \ldots, x_n)$ be $n$ elements drawn from a discrete alphabet $\mathcal{X}$. Let $N(x | x^n) = |\{i : x_i = x\}|$ be the number of times $x$ appears in the sequence $x^n$. The empirical distribution of $x^n$ is called its type: \begin{align} T_{x^n}(x) = \frac{1}{n} N(x | x^n). \end{align}

There are several definitions of typicality used in the information theory literature. Sequences that are weakly typical are close in entropy to the target distribution $P$. Sequences which are strongly typical have an empirical distribution that is close to $P$ in the $\|\cdot\|_{\infty}$ norm. Sequences which are robustly typical are those for which the ratio $T_{x^n}(x)/P(x)$ is close to $1$ in the $\|\cdot\|_{\infty}$ norm.

Weak typicality
A sequence $x^n$ is called $\epsilon$-strongly typical with respect to a distribution $P(x)$ if \begin{align} \end{align} where $H(P)$ is the entropy of $P$. For stationary and ergodic processes, weak typicality is often all that can be proved.
 * - \log T_{x^n}(x) - H(P) | \le \epsilon \qquad \forall x \in \mathcal{X},

Strong typicality
A sequence $x^n$ is called $\epsilon$-strongly typical with respect to a distribution $P(x)$ if \begin{align} \end{align} Strong typicality is often necessary for some results, especially in multiterminal information theory.
 * T_{x^n}(x) - P(x) | \le \epsilon \qquad \forall x \in \mathcal{X}.

Robust typicality
A sequence $x^n$ is called $\epsilon$-robustly typical with respect to a distribution $P(x)$ if \begin{align} \end{align}
 * T_{x^n}(x) - P(x) | \le \epsilon \cdot P(x) \qquad \forall x \in \mathcal{X}.

Conditional typicality
= The typicality of i.i.d. sequences =