Information Theory |

Entropy and Mutual Information

Fri, 08 Oct 2021 15:24:17 -0700

I’m interested in looking at some spatial mappings between pairs of cortical regions, and believe that these mappings are mediated, to some degree, by the temporal coupling between cortical areas. I don’t necessarily know the functional form of these mappings, but neurobiologically predict that these mappings are not random and have some inherent structure. I want to examine the relationship between spatial location and strength of temporal coupling. I’m going to use mutual information to measure this association.

It’s been a while since I’ve worked with information-based statistics, so I thought I’d review some proofs here.

Entropy

Given a random variable $X$, we define the entropy of $X$ as

$$\begin{align} H(X) = - \sum_{x} p(x) \cdot log(p(x)) \end{align}$$

Entropy measures the degree of uncertainty in a probability distribution. It is independent of the values $X$ takes, and is entirely dependent on the density of $X$. We can think of entropy as measuring how “peaked” a distribution is. Assume we are given a binary random variable $Y$

$$\begin{align} Y \sim Bernoulli(p) \\ \end{align}$$

such that

$$\begin{align} Y = \begin{cases} 1 & \text{with probability $p$} \\ 0 & \text{with probability $1-p$} \end{cases} \end{align}$$

If we compute $H(Y)$ as a function of $p$ and plot this result, we see the canonical curve:

Immediately evident is that the entropy curve peaks when $p=0.5$. We are entirely uncertain what value $y$ will take if we have an equal chance of sampling either 0 or 1. However, when $p = 0$ or $p=1$, we know exactly which value $y$ will take – we aren’t uncertain at all.

Entropy is naturally related to the conditional entropy. Given two variables $X$ and $Y$, conditional entropy is defined as

$$\begin{align} H(Y|X) &= -\sum_{x}\sum_{y} = p(x,y) \cdot log(\frac{p(x,y)}{p(x)}) \\ &= -\sum_{x}\sum_{y} p(y|x) \cdot p(x) \cdot log(p(y|x)) \\ &= -\sum_{x} p(x) \sum_{y} p(y|X=x) \cdot log(p(y|X=x)) \end{align}$$

where $H(Y|X=x) = -\sum_{y} p(y|X=x) \cdot log(p(y|X=x))$, the conditional entropy of $Y$ given that $X=x$. Here, we’ve used the fact that $p(x,y) = p(y|x) \cdot p(x) = p(x|y) \cdot p(y)$. To compute $H(Y|X)$, we take the weighted average of these conditional entropies, where weights are defined by the marginal probabilities of $X$.

Mutual Information

Related to entropy is the idea of mutual information. Mutual information is a measure of the mutual dependence between two variables. We can ask the following question: does knowing something about variable $X$ tell us anything about variable $Y$?

The mutual information between $X$ and $Y$ is defined as:

$$\begin{align} I(X,Y) &= \sum_{x}\sum_{y} p(x,y) \cdot log\Big(\frac{p(x,y)}{p(x) \cdot p(y)}\Big) \\ &= \sum_{x}\sum_{y}p(x,y) \cdot \log(p(x,y)) - \sum_{x}\sum_{y}p(x,y) \cdot log(p(x)) - \sum_{x}\sum_{y}p(x,y) \cdot log(p(y)) \\ &= -H(X,Y) - \sum_{x}p(x) \cdot log(p(x)) - \sum_{y}p(y) \cdot log(p(y)) \\ &= H(X) + H(Y) - H(X,Y) \end{align}$$

$I(X,Y)$ is symmetric in $X$ and $Y$:

$$\begin{align} I(X,Y) &= \sum_{x}\sum_{y} p(x,y) \cdot log\Big(\frac{p(x,y)}{p(x) \cdot p(y)}\Big) \\ &= \sum_{x}\sum_{y} p(x,y) \cdot log\Big(\frac{p(x|y)}{p(y)}\Big) \\ &= \sum_{x}\sum_{y} p(x,y) \cdot log(p(x|y)) - \sum_{x}\sum_{y}p(x,y) \cdot log(p(x)) \\ &= -H(X|Y) - \sum_{x}\sum_{y} p(x|y) \cdot p(y) \cdot log(p(x)) \\ &= -H(X|Y) - \sum_{x}log(p(x) \sum_{y}p(x|y) \cdot p(y) \\ &= -H(X|Y) - \sum_{x} p(x) \cdot log(p(x)) \\ &= H(X) - H(X|Y) \\ &= H(Y) - H(Y|X) \end{align}$$

We interpret the above to mean the following: if we are given any information about X (Y), can we reduce the uncertainty around what Y (X) should be? We understand how much variability there is in each variable independently – this is measured by the marginal entropy $H(Y)$. If knowing $X$ reduces this uncertainty, then the conditional entropy $H(Y|X)$ should be small. If knowing $X$ does not reduce this uncertainty, then $H(Y|X)$ can be at most as large as $H(Y)$, and we have learned nothing about our dependent variable $Y$.

Put another way, if $I(X,Y) = H(Y) - H(Y|X)$ is large, then the mutual information between $X$ and $Y$ is large, indicating that $X$ is informative of $Y$. However, if $I(X,Y)$ is small, then the mutual information is small, and $X$ is not informative of $Y$.

Application

For my problem, I’m given two variables, $Z$ and $C$. I’m interested in examining how knowledge of $C$ might reduce our uncertainty about $Z$. $C$ itself is defined by a pair of variables, $A$, and $B$, such that we have $C_{1} = (a_{1}, b_{1})$. $Z$ is distributed over the tensor-product space of $A$ and $B$, that is:

$$\begin{align} Z = f(A \otimes B) \end{align}$$

where $A \otimes B$ is defined as

$$\begin{align} A \otimes B = \begin{bmatrix} (a_{1},b_{1}) & \dots & (a_{1},b_{k}) \\ \vdots & \ddots & \vdots \\ (a_{p}, b_{1}) & \dots & (a_{p}, b_{k}) \end{bmatrix} \end{align}$$

such that $z_{i,j} = f(a_{i}, b_{j}$)

We define the mutual information between $Z$ and $C$ as

$$\begin{align} I(Z,C) &= \sum_{z}\sum_{c} p(z,c) \cdot log\Big(\frac{p(z,c)}{p(z)p(c)} \Big) \\ &= H(Z) - H(Z|C) \\ &= H(Z) - H(X|(A,B)) \\ &= H(Z) - \sum_{a,b} p(a,b) \sum_{z} p(z|(A,B)=(a,b)) \cdot log(p(z|(A,B)=(a,b))) \end{align}$$

where the pair $(a,b)$ represents a bin or subsection of the tensor-product space. The code for this approach can be found below:

import numpy as np

def entropy(sample):

 """
 Compute the entropy of a given data sample.
 Bins are estimated using `Freedman Diaconis` Estimator.

 Args:
 sample: float, NDarray
 data from which to compute entropy of
 Returns:
 H: entropy
 """

 if np.ndim(sample) > 1:
 sample = np.reshape(sample, np.product(sample.shape))

 edges = np.histogram_bin_edges(sample, bins='fd')
 [counts,_] = np.histogram(sample, bins=edges)

 # compute marginal distribution of sample
 m_sample = counts/counts.sum()

 return (-1)*np.nansum(m_sample*np.ma.log2(m_sample))


def mutual_information_grid(X,Y,Z):

 """
 Compute the mutual information of a dependent variable over a grid
 defined by two indepedent variables.

 Args:
 X,Y: float, NDarray
 coordinates over which dependent variable is distributed
 Z: float, array
 dependent variable
 Returns:
 estimates: dict
 keys:
 I: float
 mutual information I(Z; (X,Y))
 W: float, NDarray
 matrix of weighted conditional entropies
 marginal: float
 marginal entropy
 conditional: float
 conditional entropy
 """

 x_edges = np.histogram_bin_edges(X, bins='fd')
 [xc, _] = np.histogram(X, bins=x_edges)
 xc = xc/xc.sum()
 nx = x_edges.shape[0]-1

 y_edges = np.histogram_bin_edges(Y, bins='fd')
 [yc, _] = np.histogram(Y, bins=y_edges)
 yc = yc/yc.sum()
 ny = y_edges.shape[0]-1

 # matrix of conditional entropies for each bin
 H = np.zeros((nx, ny))

 # compute pairwise marginal probability of X/Y bins
 mxy = xc[:,None]*yc[None,:]

 for x_bin in range(nx):
 for y_bin in range(ny):

 x_idx = np.where((X>=x_edges[x_bin]) & (X<x_edges[x_bin+1]))[0]
 y_idx = np.where((Y>=y_edges[y_bin]) & (Y<y_edges[y_bin+1]))[0]

 bin_samples = Z[x_idx,:][:,y_idx]
 bin_samples = np.reshape(bin_samples,
 np.product(bin_samples.shape))

 H[x_bin, y_bin] = entropy(bin_samples)

 W = H*mxy
 conditional = np.nansum(W)
 marginal = entropy(Z)

 I = marginal - conditional

 estimates = {'mi': I,
 'weighted-conditional': W,
 'marginal': marginal,
 'conditional': conditional}

 return estimates

Cross-Entropy With Structure

Wed, 09 Dec 2020 01:12:32 -0700

As I mentioned in my previous post, I work with cortical surface segmentation data. Due to the biology of the human brain, there is considerable reproducible structure and function across individuals (thankfully!). One manifestation of this reproducibility is exemplified by the neocortex a.k.a. the thin (~2.5mm) gray matter layer of cell-bodies at the periphery of the brain. The neocortex is well known to have local homogeneity in terms of types of neuronal cells, protein and gene expression, and large-scale function, for example. Naturally, researchers have been trying to identify discrete delineations of the cortex for nearly 100 years, by looking for regions of local homogeneity of various features along the cortical manifold.

As in my previous post, I’m working on this problem using graph convolution networks (GCN). Given the logits output by a forward pass of a GCN, I want to classify a cortical node as belonging to some previously identified cortical area. Using categorical cross-entropy, we can calculate the loss of a given forward pass of the model $h(X; \Theta)$:

$$ \begin{align} L = -\sum_{k=1}^{K} \sum_{l \in \mathcal{L}} x_{l}^{k} \cdot log(\sigma(x^{k})_{l}) \end{align} $$

where $x^{k}$ is the output of the model for a single node, $x_{l}^{k}$ is the one-hot-encoding value of the true labels, and $\sigma$ is the softmax function. Importantly, the cross-entropy cost is high when the probability assigned to the true label of a node is small i.e. $log(0) = \infty$, while $log(1) = 0$ – as such, the cross-entropy tries to minimize the rate of false negatives.

However, we can incorporate more structure into this loss function. As I mentioned previously, we know that the brain is highly reproducible across individuals. In our case, we have years of biological evidence pointing to the fact that functional brain areas i.e. like the primary visual area (V1), will always be in the same anatomical location i.e. posterior occipital cortex – and will always be adjacent to a small subset of other functionally-defined areas, like the secondary visual area (V2), for example.

Various maps of the primate visual cortex. Tootell et al, 2003.

This leads us to the idea of assigning a high cost when nodes which should be in V1, for example, are assigned labels of regions that are not adjacent to V1. We do so by defining another cost function:

\begin{align} G = -\sum_{k=1}^{k}\sum_{l \in \mathcal{L}} \sum_{h \in \mathcal{L} \setminus \mathcal{N_{l}}} w_{l}^{k} \cdot log(1-\sigma(x^{k})_{l}) \end{align}

where $w_{l}^{k}$ is the probability weight assigned to label $h \in \mathcal{L}\setminus \mathcal{N_{l}}$ i.e. the set of labels not adjacent to label $l$. In order to follow the idea of a cross-entropy, we enforce the following constraints on weights $\mathbf{w}$:

$$ \begin{align} w_{l}^{k} >&= 0 \\ \sum_{l \in \mathcal{L}} w_{l}^{k} &= 1 \end{align} $$

such that the vector $\mathbf{w}$ is a probability distribution over labels. Importantly, if we consider more closely what this loss-function is doing, we are encouraging the predicted label of $x^{k}$ to not be in the set $\mathcal{L} \setminus \mathcal{N_{l}}$. Assume, for example, that the true label of $x^{k}$ is $t$, and that label $j$ is not adjacent to label $t$ on the cortical surface. If the softmax function assigns a probability $p(x^{k}{l} = j) = 0.05$, then $log(1-p(x^{k}{l} = j))$ will be small. However, if $p(x^{k}{l} = j) = 0.95$, then $log(1-p(x^{k}{l} = j))$ will be large. Consequently, we penalize higher probabilities assigned to labels not adjacent to our true label – i.e. ones that are not even biologically plausible. If a candidate label of $x^{k}{l} \in \mathcal{N{t}}$, we simply set $w_{l}^{k} = 0$ – that is, we do not penalize the true label (obviously), or labels adjacent to the true label, since these are the regions we really want to consider.

Below, I’ve implemented this loss function using Pytorch and Deep Graph Library. Assume that we are given the adjacency matrix of our mesh, the logits of our model, and the true label of our training data:

import numpy as np

import dgl
import dgl.function as fn
import torch.nn.functional as F

import torch as th

def structured_cross_entropy(graph, logits, target):

 """
 Compute a structured cross-entropy loss.

 Loss penalizes high logit probabilities assigned to labels
 that are not directly adjacent to the true label.

 Parameters:
 - - - - -
 graph: DGL graph
 input graph structure
 input: torch tensor
 logits from model
 target: torch tensor
 true node labeling
 Returns:
 - - - -
 loss: torch tensor
 structured cross-entropy loss
 """

 # compute one-hot encoding of true labels
 hot_encoding = F.one_hot(target).float()

 # identify adjacent labels
 weight = th.matmul(hot_encoding.t(),
 th.matmul(graph.adjacency_matrix(), hot_encoding))
 weight = (1-(weight>0).float())

 # compute inverted encoding (non-adjacent labels receive value of 1)
 inv_encoding = weight[target]
 # weight by 1/(# non adjacent)
 # all non-adjacent labels receive the same probability
 # adjacent labels and self-label receive probability of 0
 inv_encoding = inv_encoding / inv_encoding.sum(1).unsqueeze(1)
 loss = th.sum(inv_encoding*th.log(1-F.softmax(logits)), 1)

 return -loss.mean()

If we wanted to use this loss function in conjunction with another loss, like the usual cross-entropy, we could perform something like the following:


# define a regularizing parameter
gamma = 0.1
# define the usual cross-entropy loss function
loss_fcn = torch.nn.CrossEntropyLoss()
loss = loss_function(logits, target) + gamma*structured_cross_entropy(graph, logits, target)

# because our new loss functions performs computations using Pytorch
# the computation history is stored, and we can compute the gradient 
# with respect to this combined loss as

optimizer.zero_grad() # zero the gradients (no history)
loss.backward() # compute new gradients
optimizer.step() # update weights and parameters w.r.t new gradient

Because we’re optimizing two loss functions now i.e. the global accuracy of the model as defined using the conventional cross-entropy, and the desire for predicted labels to not be far away from the true label using the structured cross-entropy, this combination of loss functions will likely have the effect of slightly reducing global accuracy – however, it will have the effect of generating predictions showing fewer anatomically spurious labels i.e. we are less likely to see vertices in the frontal lobe labeled as V1, or vertices in the lateral parietal cortex labeled as Anterior Cingulate. Global predictions will be more biologically plausible. While GCNs as a whole are already better able to incorporate local spatial information than other models due to the fact that they convolve signals based on the adjacency structure of the network in question, I have found empirically that these anatomically spurious predictions are still possible – hence the need for this more-structured regularization.