Slepian-Wolf network

Motivation
One of the canonical problems in multi-terminal information theory is the encoding of correlated sources using distributed encoders. A simple example is illustrated in the figure to the left. Here we have have two correlated sources $X_1$ and $X_2$ distributed according to some joint distribution $p_{X_1,X_2}\left(X_1,X_2\right)$. Each source is only separately available at one of two encoders. The job of the two encoders is to encode their sources such that the centralized decoder can 'losslessly recover both sources.



A natural question one might ask is : "What are the information theoretic bounds on the set of rate pairs $(R_1, R_2)$ such that the decoder recovers the source sequence with an arbitrarily small probability of error?" As a first step, we might consider the encoding of the two sources independently; i.e., each encoder uses a rate equal to the marginal entropy of its source. This source coding theorem tells us that this rate is sufficient for lossless recovery of each source at the central decoder. But it does not exploit the mutual information common to both sources. On the other hand, a centralized encoder could use a rate equal to the joint entropy for lossless recovery of the sources at the decoder. With the exception of independent sources, the joint entropy is less than the sum of the two marginal entropies. It might appear then, at first glance, that distributed source coding is somehow inefficient as compared to centralized encoding. However, the theorem of David Slepian and Jack Wolf tells us that distributed source coding can be done with a sum rate ($R_1 + R_2$) equal to the joint entropy.

Two Sources
We have two encoder maps \begin{align} f_1 &: \mathcal{X}_1^n \rightarrow \{1, 2, \ldots, 2^{nR_1}\},\\ f_2 &: \mathcal{X}_2^n \rightarrow \{1, 2, \ldots, 2^{nR_2}\} \end{align} and a single decoder map \begin{equation} g : \{1, 2, \ldots, 2^{nR_1}\} \times \{1, 2, \ldots, 2^{nR_2}\} \rightarrow \mathcal{X}_1^n \times \mathcal{X}_2^n. \end{equation} The probability of error is given by \begin{equation} P_e^{(n)} = Pr\left\{\left(X_1^n, X_2^n\right) \neq g\left(f_1\left(X_1^n\right), f_2\left(X_2^n\right)\right)\right\} \end{equation}

Let $(X_{1}, X_{2})$ be i.i.d $\sim p_{X_{1},X_{2}}\left(X_{1},X_{2}\right)$. The set of rate vectors achievable for distributed source coding with seperate encoders and a common decoder is defined by \begin{align} R_{1} &\geq H\left(X_1|X_2\right)\\ R_{2} &\geq H\left(X_2|X_1\right)\\ R_{1}+R_{2} &\geq H\left(X_1, X_2\right) \end{align}



Achievability
The two key components to the achievability of Slepian-Wolf are random binning and joint typicality.

Converse
The key component to the converse of Slepian-Wolf is Fano's inequality

Connections with Graph Coloring
For distributed source coding of correlated sources, we can encode $X_{1}$ efficiently (i.e., with probability of error going to zero in the block length) using $nH\left(X_{1}\right)$ bits. Associated with every sequence $X_{1}^n$ is a typical fan $X_{2}^n$ that are jointly typical with given $X_{1}^n$. Formally, \begin{equation} A_{\epsilon,x_{1}^n}^{n}\left(X_{2}^n\right) = \left\{X_{2}^n \left| \left(x_{1}^n, X_{2}^n\right) \in A_{\epsilon}^n(X_{1}^n, X_{2}^n)\right.\right\}. \end{equation} If the encoder for $X_{2}^n$ knew $X_{1}^n$, then it could just send the index of $X_{2}^n$ within the typical fan. The decoder then could use $X_{1}^n$ to identify the typical fan and the index from the second encoder to identify the particular $X_{2}$ within this typical fan. However, the second encoder doesn't have the $X_{1}^n$ available. Instead, it will color all sequence $X_{2}^n$ with $2^{nR_2}$ colors. With a sufficiently high number of colors, the properties of joint typicality ensure that with high probability all of the colors in a particular fan will be a different and the color of the $X_{2}^n$ sequence uniquely defines the sequence within the typical fan.

Multiple Sources
Let $\left(X_1, X_2, \ldots, X_{n}\right)$ be i.i.d $\sim p\left(X_{1}, X_{2}, \ldots, X_{n}\right)$. Then the set of rate vectors achievable for distributed source coding with separate encoders and a common decoder is defined by \begin{equation} R(S) > H(X(S)|X(S^c)) \end{equation} for all $S \subseteq \{1, 2, \ldots, |V|-1\}$, where \begin{equation} R(S) = \sum_{i=1}^{n}R_i \end{equation} and $X(S) = \{X_j: j \in S\}$. The proof of Slepain-Wolf for multiple sources naturally extends from the proof for two sources.