Channel coding theorem

Theorem For a discrete memoryless channel with probability $$p(y|x)$$ all rates below $$C \triangleq max_{p(x)} I(y;x)$$ are achievable. That is $$\forall R<C, \exists$$ a code sequence $$(2^{nR}, n)$$, such that $$\lim_{n \rightarrow \infty} = 0$$.

Proof outline For a fixed $$p(x)$$ generate $$2^{nR}$$ independent codewords according to the distribution $$p(x^n) = \prod_{i=1}^n p(x_i)$$. Choose a message $$W$$ according to a uniform distribution $$P ( W = w ) = 2^{-nR}, w = 1, 2, \ldots, 2^{nR}$$. Transmit the $$w$$-th codeword over the channel. On the receiver side, conclude that codeword $$v$$ was sent if the received codeword (which has the distribution $$P(y^n | x^n(w)) = \prod_{i=1}^n p(y_i|x_i(w))$$) is jointly typical with the $$v$$-th codeword and is not jointly typical with any other codeword.

Converse If for a code sequence $$(2^{nR},n)$$ with $$\lim_{n \rightarrow \infty} = 0$$, $$R < C$$.

Proof