Huffman codes

Huffman coding provides a practical scheme to achieve optimal length for data compression. For any compression scheme the achievable average length is lower bounded by the entropy $ H(X) \le L$. The Huffman algorithm satisfies this equation with equality for dyadic probability distributions. The algorithm is as follows. Consider a discrete PMF with probabilities $p=\{ p_{k} | k=1,..,n\}$ and corresponding symbols $\{X_{k} | k = 1,..n \}$, WLOG assume $p_{i} \le p_{j}$ for $i \le j$. Label the two least likely symbols using a binary representation, $X_{1}=0$ and $X_{2}=1$. Combine their corresponding probabilities $p_{1,2} = p_{1} + p_{2}$ and create a parent symbol $X_{1,2}$. Create a new PMF $\hat{p}=\{p_{1,2},p_{3},...,p_{n} \}$ and a new symbol set $\{X_{1,2},X_{3},...,X_{n} \}$. Rearrange and order into $\hat{p}=\{\hat{p_{k}}|k=1,..,n-1\}$ such that $\hat{p_{i}} \le \hat{p_{j}}$ for $i \le j$ with new symbol set $\{\hat{X_{k}}|k=1,..,n-1\}$. Again combine the lowest probabilities and label their corresponding symbols, with propagation and concatenation for parent nodes. For example if $X_{1,2}$ is one of the two least likely symbols a binary label will be propagated down to both $X_{1}$ and $X_{2}$ giving $X_{1}=00$, $X_{2}=01$ if $X_{1,2}$ is the least likely symbol and $X_{1}=10$, $X_{2}=11$ if $X_{1,2}$ is the second least likely symbol. Repeat this process until $\hat{p} = \{1\}$. This procedure can be modified for an alphabet of size m, i.e. for $m=4$, four symbols will be grouped together for each iteration and each symbol grouped will be labeled with either 0,1,2, or 3 based on their probability, i.e. $X_{1}=3012$.

Simple Example
Consider the PMF $p=\{\frac{1}{6},\frac{1}{3},\frac{1}{2} \}$ and corresponding symbol set $ \{ X_{1}, X_{2}, X_{3} \}$. The iterations will be as follows

Iteration 1
$$X_{1}=0 \quad X_{2}=1$$ $$\hat{p}=\{\frac{1}{2} ,\frac{1}{2}\}$$ $$\{X_{1,2}, X_{3} \}$$ $$\hat{p} \ne \{1\}$$

Iteration 2
$$X_{1} = 00 \quad X_{2}=01 \quad X_{3}=1$$ $$\hat{p}=\{1\}$$ $$\{X_{1,2,3}\}$$ $$\hat{p}=\{1\}$$