Mathematics |

Entropy and Mutual Information

Fri, 08 Oct 2021 15:24:17 -0700

I’m interested in looking at some spatial mappings between pairs of cortical regions, and believe that these mappings are mediated, to some degree, by the temporal coupling between cortical areas. I don’t necessarily know the functional form of these mappings, but neurobiologically predict that these mappings are not random and have some inherent structure. I want to examine the relationship between spatial location and strength of temporal coupling. I’m going to use mutual information to measure this association.

It’s been a while since I’ve worked with information-based statistics, so I thought I’d review some proofs here.

Entropy

Given a random variable $X$, we define the entropy of $X$ as

$$\begin{align} H(X) = - \sum_{x} p(x) \cdot log(p(x)) \end{align}$$

Entropy measures the degree of uncertainty in a probability distribution. It is independent of the values $X$ takes, and is entirely dependent on the density of $X$. We can think of entropy as measuring how “peaked” a distribution is. Assume we are given a binary random variable $Y$

$$\begin{align} Y \sim Bernoulli(p) \\ \end{align}$$

such that

$$\begin{align} Y = \begin{cases} 1 & \text{with probability $p$} \\ 0 & \text{with probability $1-p$} \end{cases} \end{align}$$

If we compute $H(Y)$ as a function of $p$ and plot this result, we see the canonical curve:

Immediately evident is that the entropy curve peaks when $p=0.5$. We are entirely uncertain what value $y$ will take if we have an equal chance of sampling either 0 or 1. However, when $p = 0$ or $p=1$, we know exactly which value $y$ will take – we aren’t uncertain at all.

Entropy is naturally related to the conditional entropy. Given two variables $X$ and $Y$, conditional entropy is defined as

$$\begin{align} H(Y|X) &= -\sum_{x}\sum_{y} = p(x,y) \cdot log(\frac{p(x,y)}{p(x)}) \\ &= -\sum_{x}\sum_{y} p(y|x) \cdot p(x) \cdot log(p(y|x)) \\ &= -\sum_{x} p(x) \sum_{y} p(y|X=x) \cdot log(p(y|X=x)) \end{align}$$

where $H(Y|X=x) = -\sum_{y} p(y|X=x) \cdot log(p(y|X=x))$, the conditional entropy of $Y$ given that $X=x$. Here, we’ve used the fact that $p(x,y) = p(y|x) \cdot p(x) = p(x|y) \cdot p(y)$. To compute $H(Y|X)$, we take the weighted average of these conditional entropies, where weights are defined by the marginal probabilities of $X$.

Mutual Information

Related to entropy is the idea of mutual information. Mutual information is a measure of the mutual dependence between two variables. We can ask the following question: does knowing something about variable $X$ tell us anything about variable $Y$?

The mutual information between $X$ and $Y$ is defined as:

$$\begin{align} I(X,Y) &= \sum_{x}\sum_{y} p(x,y) \cdot log\Big(\frac{p(x,y)}{p(x) \cdot p(y)}\Big) \\ &= \sum_{x}\sum_{y}p(x,y) \cdot \log(p(x,y)) - \sum_{x}\sum_{y}p(x,y) \cdot log(p(x)) - \sum_{x}\sum_{y}p(x,y) \cdot log(p(y)) \\ &= -H(X,Y) - \sum_{x}p(x) \cdot log(p(x)) - \sum_{y}p(y) \cdot log(p(y)) \\ &= H(X) + H(Y) - H(X,Y) \end{align}$$

$I(X,Y)$ is symmetric in $X$ and $Y$:

$$\begin{align} I(X,Y) &= \sum_{x}\sum_{y} p(x,y) \cdot log\Big(\frac{p(x,y)}{p(x) \cdot p(y)}\Big) \\ &= \sum_{x}\sum_{y} p(x,y) \cdot log\Big(\frac{p(x|y)}{p(y)}\Big) \\ &= \sum_{x}\sum_{y} p(x,y) \cdot log(p(x|y)) - \sum_{x}\sum_{y}p(x,y) \cdot log(p(x)) \\ &= -H(X|Y) - \sum_{x}\sum_{y} p(x|y) \cdot p(y) \cdot log(p(x)) \\ &= -H(X|Y) - \sum_{x}log(p(x) \sum_{y}p(x|y) \cdot p(y) \\ &= -H(X|Y) - \sum_{x} p(x) \cdot log(p(x)) \\ &= H(X) - H(X|Y) \\ &= H(Y) - H(Y|X) \end{align}$$

We interpret the above to mean the following: if we are given any information about X (Y), can we reduce the uncertainty around what Y (X) should be? We understand how much variability there is in each variable independently – this is measured by the marginal entropy $H(Y)$. If knowing $X$ reduces this uncertainty, then the conditional entropy $H(Y|X)$ should be small. If knowing $X$ does not reduce this uncertainty, then $H(Y|X)$ can be at most as large as $H(Y)$, and we have learned nothing about our dependent variable $Y$.

Put another way, if $I(X,Y) = H(Y) - H(Y|X)$ is large, then the mutual information between $X$ and $Y$ is large, indicating that $X$ is informative of $Y$. However, if $I(X,Y)$ is small, then the mutual information is small, and $X$ is not informative of $Y$.

Application

For my problem, I’m given two variables, $Z$ and $C$. I’m interested in examining how knowledge of $C$ might reduce our uncertainty about $Z$. $C$ itself is defined by a pair of variables, $A$, and $B$, such that we have $C_{1} = (a_{1}, b_{1})$. $Z$ is distributed over the tensor-product space of $A$ and $B$, that is:

$$\begin{align} Z = f(A \otimes B) \end{align}$$

where $A \otimes B$ is defined as

$$\begin{align} A \otimes B = \begin{bmatrix} (a_{1},b_{1}) & \dots & (a_{1},b_{k}) \\ \vdots & \ddots & \vdots \\ (a_{p}, b_{1}) & \dots & (a_{p}, b_{k}) \end{bmatrix} \end{align}$$

such that $z_{i,j} = f(a_{i}, b_{j}$)

We define the mutual information between $Z$ and $C$ as

$$\begin{align} I(Z,C) &= \sum_{z}\sum_{c} p(z,c) \cdot log\Big(\frac{p(z,c)}{p(z)p(c)} \Big) \\ &= H(Z) - H(Z|C) \\ &= H(Z) - H(X|(A,B)) \\ &= H(Z) - \sum_{a,b} p(a,b) \sum_{z} p(z|(A,B)=(a,b)) \cdot log(p(z|(A,B)=(a,b))) \end{align}$$

where the pair $(a,b)$ represents a bin or subsection of the tensor-product space. The code for this approach can be found below:

import numpy as np

def entropy(sample):

 """
 Compute the entropy of a given data sample.
 Bins are estimated using `Freedman Diaconis` Estimator.

 Args:
 sample: float, NDarray
 data from which to compute entropy of
 Returns:
 H: entropy
 """

 if np.ndim(sample) > 1:
 sample = np.reshape(sample, np.product(sample.shape))

 edges = np.histogram_bin_edges(sample, bins='fd')
 [counts,_] = np.histogram(sample, bins=edges)

 # compute marginal distribution of sample
 m_sample = counts/counts.sum()

 return (-1)*np.nansum(m_sample*np.ma.log2(m_sample))


def mutual_information_grid(X,Y,Z):

 """
 Compute the mutual information of a dependent variable over a grid
 defined by two indepedent variables.

 Args:
 X,Y: float, NDarray
 coordinates over which dependent variable is distributed
 Z: float, array
 dependent variable
 Returns:
 estimates: dict
 keys:
 I: float
 mutual information I(Z; (X,Y))
 W: float, NDarray
 matrix of weighted conditional entropies
 marginal: float
 marginal entropy
 conditional: float
 conditional entropy
 """

 x_edges = np.histogram_bin_edges(X, bins='fd')
 [xc, _] = np.histogram(X, bins=x_edges)
 xc = xc/xc.sum()
 nx = x_edges.shape[0]-1

 y_edges = np.histogram_bin_edges(Y, bins='fd')
 [yc, _] = np.histogram(Y, bins=y_edges)
 yc = yc/yc.sum()
 ny = y_edges.shape[0]-1

 # matrix of conditional entropies for each bin
 H = np.zeros((nx, ny))

 # compute pairwise marginal probability of X/Y bins
 mxy = xc[:,None]*yc[None,:]

 for x_bin in range(nx):
 for y_bin in range(ny):

 x_idx = np.where((X>=x_edges[x_bin]) & (X<x_edges[x_bin+1]))[0]
 y_idx = np.where((Y>=y_edges[y_bin]) & (Y<y_edges[y_bin+1]))[0]

 bin_samples = Z[x_idx,:][:,y_idx]
 bin_samples = np.reshape(bin_samples,
 np.product(bin_samples.shape))

 H[x_bin, y_bin] = entropy(bin_samples)

 W = H*mxy
 conditional = np.nansum(W)
 marginal = entropy(Z)

 I = marginal - conditional

 estimates = {'mi': I,
 'weighted-conditional': W,
 'marginal': marginal,
 'conditional': conditional}

 return estimates

Distances Between Subspaces

Wed, 24 Jun 2020 15:27:11 -0700

I’m working with some multi-dimensional float-valued data – I’ll call a single instance of this data $X \in \mathbb{R}^{n \times k}$. I have multiple samples $X_{1}, X_{2}…X_{t}$, and want to compare these subspaces – namely, I want to compute the distance between pairs of subspaces.

Let’s assume that our subspaces are not rank-deficient – i.e. for a given subspace sample, all of our dimensions are linearly independent. Thus, the $k$ vectors form a basis set that spans some $k$-d subspace in $\mathbb{R}^{n}$. We can think of each $k$-d subspace as a hyperplane in $(k+1)$-d space, just as we can think of a 2-d plane in 3-d space. One way to compare these subspaces is by using the “principal angles between subspaces” (or angles between flats). We can compare the “angles” between these hyperplanes, which will tell us how “far apart” the two subspaces are.

Intersecting 2D linear subspaces.

This comparison is effectively based on the QR decomposition and the Singular Value Decomposition. For two subspaces $[U, W]$, we compute the QR decomposition of both:

$$\begin{align} U &= Q_{u}R_{u}\\ W &= Q_{w}R_{w}\\ \end{align}$$

where $Q_{u}$ and $Q_{w} \in \mathbb{R}^{n \times k}$ are orthonormal bases such that $Q_{u}^{T}Q_{u} = Q_{w}^{T}Q_{w} = I_{k}$ that span the same subspace as the original columns of $U$ and $W$, and $R_{u}$ and $R_{w} \in \mathbb{R}^{k \times k}$ are upper triangular matrices. Next, we compute the matrix $D = \langle Q_{u}, Q_{w} \rangle = Q_{u}^{T} Q_{w} \in \mathbb{R}^{k \times k}$, and then apply the singular value decomposition:

$$\begin{align} D = USV^{T} \end{align}$$

We can sort of think of $D$ as the cross-covariance matrix. As such, the singular vectors represent the main orthogonal axes of cross-covariation between our two subspaces, while the singular values represent angles. In order to compute the principal angles of our subspaces, we simply take

$$\begin{align} \theta &= cos^{-1}(S) \\ &=cos^{-1}[\sigma_{1}, \sigma_{2}…\sigma_{k}] \end{align}$$

which gives us the principal angles (in radians). Because the SVD is invariant to sign (+/-), the principal angles range between $\Big[0, \frac{\pi}{2}\Big]$. This means that subspaces that span the same space have a principal angle of 0, and subspaces that are orthogonal (maximally far apart) to one another have a principal angle of $\frac{\pi}{2}$.

In order to compute the “distance” between our subspaces, we can apply various metrics to our vector of principal angles. The simplest approach is to apply the $L2$ norm to our vector of principal angles, $\theta$, as

$$\begin{align} d(X_{i}, X_{j}) = \sqrt{\sum_{n=1}^{k} cos^{-1}(\sigma_{n})^{2}} \end{align}$$

This metric is called the Grassmann Distance and is formally related to the geodesic distance between subspaces distributed on the Grassmannian manifold.

Grassmann manifold and its tangent space.

This, however, is a topic for another future blog post. There are a variety of metrics we can use to compute the pairwise distance between subspaces, some of which are

Asimov: $\; max(\theta)$
Fubini-Study: $\; cos^{-1}(\prod sin(\theta))$
Spectral: $\; 2 sin(\frac{max(\theta)}{2})$

but all are fundamentally based on some function of our vector of principal angles, $\theta$.

Dose-Response Curves and Biomarker Diagnostic Power

Tue, 27 Aug 2019 11:12:32 -0700

The other day, one of my friends and colleagues (I’ll refer to him as “Dr. A”) asked me if I knew anything about assessing biomarker diagnostic power. He went on to describe his clinical problem, which I’ll try to recount here (but will likely mess up some of the relevant detail – his research pertains to generating induced pluripotent cardiac stem cells, which I have little to no experience with):

“So chemotherapy is meant to combat cancer. But some clinicians, and their patients, have found that some forms of chemotherapy and anti-cancer drugs later result in problems with the heart and vasculature – these problems are collectively referred to as ‘cardiotoxicity’.

We’re interested in developing biomarkers that will help us identify which patients might be susceptible to cardiotoxicity, and in assessing the predictive power of these biomarkers. Can you help me?”

What follows will be my exploration into what Dr. A called dose-response curves, and my approach on how to use these curves to assess biomarker diagnostic power. I’ll do a walkthrough of some Python code that I’ve written up, where I’ll examine dose-response curves and their diagnostic power using the receiver operating characteristic ( ROC) and the closely related area under the curve ( AUC) metrics.

If you want to see the actual dose-response curve analysis, skip to the Analyzing Synthetic Dose-Response Curves section down below.

Brief Introduction to the ROC and AUC

I first wanted to describe to Dr. A how to use the ROC for biomarkers, so I made a brief Python tutorial for him to look at. Below begins a more in-depth description of what I sent him.

We start by importing some necessary libraries:

%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib.lines import Line2D

import numpy as np
from sklearn.metrics import auc

Next, we define a function to compute the ROC curve. While scikit-learn has a function to compute the ROC, computing it yourself makes it easier to understand what the ROC curve actually represents.

The ROC curve is essentially a curve that plots the true positive and false positive classification rates against one another, for various user-defined thresholds of the data. We pick a biomarker level threshold – let’s say $T$ – assign each sample a value of 0 or 1 depending on whether its measured biomarker level is greater than or less than $T$, and then compare these assignments to the true Case/Control classifications to compute the true and false positive rates. We plot these rates for many $T$ to generate the ROC curve.

The ROC lets you look at how sensitive your model is to various parameterizations – are you able to accurately identify Cases from Controls? How acceptable are misclassified results? In this situation, we don’t want to classify someone as healthy when in fact they might develop cardiotoxicity, so we want to allow some flexibility in terms of the number of false positives generated by our model – it’s a classic case of “better safe than sorry”, since the “sorry” outcome might be an accidental patient death.

def roc(case, control, npoints, gte=True):

 """
 Compute ROC curve for given set of Case/Control samples.

 Parameters:
 - - - - -
 case: float, array
 Samples from case patients
 control: float, array
 Samples from control patients
 npoints: int
 Number of TP/FP pairs to generate
 gte: boolean
 Whether control mean expected to be greater
 than case mean.

 Returns:
 - - - -
 specificity: float, array
 false positive rates
 sensitivity: float, array
 true positive rates
 """

 # make sure generating more
 # than 1 TP/FP pair
 # so we can plot an actual ROC curve
 assert npoints > 1


 # made sure case and control samples
 # are numpy arrays
 case = np.asarray(case)
 control = np.asarray(control)

 # check for NaN values
 # keep only indices without NaN
 case_nans = np.isnan(case)
 cont_nans = np.isnan(control)
 nans = (case_nans + cont_nans)

 specificity = []
 sensitivity = []

 # we'll define the min and max thresholds 
 # based on the min and max of our data
 conc = np.concatenate([case[~nans],control[~nans]])

 # function comparison map
 # use ```gte``` parameter
 # if we expect controls to be less than
 # cases, gte = False
 # otherwise gte = True
 comp_map = {'False': np.less,
 'True': np.greater}

 # generate npoints equally spaced threshold values
 # compute the false positive / true positive rates 
 # at each threshold
 for thresh in np.linspace(conc.min(), conc.max(), npoints):

 fp = (comp_map[gte](case[~nans], thresh)).mean()
 tn = 1-fp

 tp = (comp_map[gte](control[~nans], thresh)).mean()
 fn = 1-tp

 specificity.append(tn)
 sensitivity.append(tp)

 return [np.asarray(specificity), np.asarray(sensitivity)]

Next up, I generate 5 different datasets. Each dataset corresponds to fake samples from a “Control” distribution, and a “Case” distribution – each is distributed according to a univariate Normal distribution. The Control distribution remains the same in each scenario: $N(\mu = 10, \sigma = 1)$, but I change the $\mu_{Case}$ parameter of the Case distribution in each instance, such that $\mu_{Case} \in [5,6,7,8,9,10]$.

We plot the example datasets as follows, and then compute the ROC curves for each dataset.

# define unique means
m1 = np.arange(5,11)
samples = {}.fromkeys(m1)

# define control distribution (this stays the same across 
n2 = np.random.normal(loc=10, scale=1, size=2000)

fig, (ax) = plt.subplots(2, 3, figsize=(15, 6))

for i, ax in enumerate(fig.axes):

 n1 = np.random.normal(loc=m1[i], scale=1, size=2000)
 samples[m1[i]] = n1

 ax.hist(n1, 25, alpha=0.5, label='Case', density=True,)
 ax.hist(n2, 25, alpha=0.5, label='Control', density=True,)
 ax.set_title('Mean: {:}, Sigma: 1'.format(m1[i]), fontsize=15)
 ax.set_xlabel('Biomarker Measures Variable', fontsize=15)
 ax.set_ylabel('Density', fontsize=15)
 ax.legend()

plt.tight_layout()

Example Case/Control datasets.

fig, (ax1) = plt.subplots(1, 1)
for mean, case_data in samples.items():

 [spec, sens] = roc(case_data, n2, 100, gte=True)
 A = auc(spec, sens)

 L = 'Mean: %i, AUC: %.3f' % (mean, A)
 plt.plot(1-np.asarray(spec), np.asarray(sens), label=L)
 plt.legend(bbox_to_anchor=(1.04,1), fontsize=15)
 plt.xlabel('1-Specificity', fontsize=15)
 plt.ylabel('Sensitivity', fontsize=15)

 plt.title('ROC for Varying Case Distributions', fontsize=15)

ROC curves for example datasets.

We see that, as the distributions become less separated, the ability to distinguish points from either distribution is diminished. This is shown by 1) a flattening of the ROC curve towards the diagonal, along with 2) the integration of the ROC curve, which generates the AUC metric. When the distributions are far apart (as when the $\mu_{Case} = 5$), it is quite easy for a simple model to distinguish points sampled from either distribution, meaning this hypothetical model has good diagnostic power.

Analyzing Synthetic Dose-Response Curves

We now analyze some synthetic data, generated to look like dose-response curves. As a refresher, dose-response curves measure the behavior of some tissue cells in response to increasing levels of (generally) drugs. The $x$-axis is the drug dose, measured in some concentration or volume, and the $y$-axis is generally some measure of cell death or survival, generally ranging from 0% to 100%. The curves, however, look sigmoidal. So let’s first generate a function to create sigmoid curves.

def sigmoid(beta, intercept, x):

 """
 Fake sigmoid function, takes in coefficient, shift, and dose values.

 Parameters:
 - - - - -
 beta: float
 slope
 intercept: float
 negative exponential intercept
 x: float, array
 data samples

 Returns:
 - - - -
 dose_response: float, array
 single-subject dose-response vector
 Between 0 and 1.
 """

 dose_resonse = 1 / (1 + np.exp(-beta*x + intercept))

 return dose_resonse

Next, let’s actually generate some synthetic data for a dataset of fake subjects. I want to incorporate some variability into the Cases and Controls, so I’ll sample the subject parameters from distributions. In this case, for each subject, I’ll sample the logistic curve slope coefficient from $Beta$ distributions, and the intercept from $Normal$ distributions. We’ll sample 1000 Cases, and 1000 Controls.

For the slopes, we have

$$ \begin{align} \beta_{Control} &\sim Beta(a=5, b=3) \ \beta_{Case} &\sim Beta(a=10, b=2) \end{align} $$

and for the intercepts, we have

$$ \begin{align} I_{Control} &\sim Normal(\mu=0, \sigma=1) \ I_{Case} &\sim Normal(\mu=4, \sigma=1) \end{align} $$

As such, the dose-response curve for individual, $k$, is generated as follows:

$$ dr_{k} = \frac{1}{1 + e^{-(\beta_{k}X + I_{k})}} $$

where $\beta_{k}$ and $I_{k}$ are the slope and intercept values for the given subject. These distributional parameterizations are arbitrary – I just wanted to be able to incorporate variability across subjects and groups.

Let’s generate some random Case/Control dose-response data and plot the coefficient histograms:

# n cases and controls
S = 1000

# dictionary of slopes and intercept values for each subject
controls = {k: {'beta': None,
 'intercept': None} for k in np.arange(S)}
cases = {k: {'beta': None,
 'intercept': None} for k in np.arange(S)}

# get lists of betas and intercepts
beta_control = []
beta_case = []

intercept_control = []
intercept_case = []

for i in np.arange(S):

 controls[i]['beta'] = np.random.beta(a=5, b=3)
 controls[i]['intercept'] = np.random.normal(loc=0, scale=1)

 intercept_control.append(controls[i]['intercept'])
 beta_control.append(controls[i]['beta'])

 cases[i]['beta'] = np.random.beta(a=10, b=2)
 cases[i]['intercept'] = np.random.normal(loc=4, scale=1)

 intercept_case.append(cases[i]['intercept'])
 beta_case.append(cases[i]['beta'])

Intercept histograms look like two different $$Normal$$ distributions:

fig = plt.figure(figsize=(12, 8))
plt.hist(intercept_control, 50, color='r', alpha=0.5, density=True, label='Control');
plt.hist(intercept_case, 50, color='b', alpha=0.5, density=True, label='Case');
plt.ylabel('Density', fontsize=15);
plt.xlabel('Intercepts', fontsize=15);
plt.title('Intercepts Coefficients By Group', fontsize=15);
plt.legend(fontsize=15);
plt.tight_layout()

Intercept distributions for synthetic dose-response curves.

Slope histograms look like two different $Beta$ distributions:

fig = plt.figure(figsize=(12, 8))
plt.hist(beta_control, 50, color='r', alpha=0.5, label='Control', density=True);
plt.hist(beta_case, 50, color='b', alpha=0.5, label='Case', density=True);
plt.ylabel('Density', fontsize=15);
plt.xlabel('Betas', fontsize=15);
plt.title('Slope Coefficients By Group', fontsize=15);
plt.legend(fontsize=15);
plt.tight_layout()

Slope distributions for synthetic dose-response curves.

Now we’ll generate some fake dose-response curves for each of the 1000 Controls, and 1000 Cases. We’ll plot a subset of these curves to visualize our cross-group curve variability.

# define synthetic dose range
doses = np.linspace(-15,15,500)
dose_min = doses.min()
shifted_dose = doses + np.abs(dose_min)

fig, (ax1) = plt.subplots(figsize=(12, 8))

ec50_control = []
ec50_case = []

for c in np.arange(S):

 control_sample = sigmoid(controls[c]['beta'], controls[c]['intercept'], doses)
 case_sample = sigmoid(cases[c]['beta'], cases[c]['intercept'], doses)

 ec50_control.append(shifted_dose[control_sample < 0.5].max())
 ec50_case.append(shifted_dose[case_sample < 0.5].max())

 if (c % 15) == 0:

 ax1.plot(shifted_dose, control_sample, c='r', linewidth=3, alpha=0.3)
 ax1.plot(shifted_dose, case_sample, c='b', linewidth=3, alpha=0.3)

plt.legend({})
plt.title('Dose Response Curve', fontsize=20);
plt.xlabel('Dose', fontsize=20);
plt.xticks(fontsize=15)
plt.ylabel('Response', fontsize=20);
plt.yticks(fontsize=15)

custom_lines = [Line2D([0], [0], color='r', lw=4),
 Line2D([0], [0], color='b', lw=4)]
plt.legend(custom_lines, ['Control', 'Case'], fontsize=20);

Case/Control dose-response curves.

For our preliminary biomarker of interest, let’s look at the ec50, which is the dose at which 50% of the cells show some response (i.e. where our $y$-axis = 0.5), for each sample in our dataset. We’ll plot these doses as a function of Cases and Controls.

fig = plt.figure(figsize=(12, 8))
plt.hist(ec50_control, 50, color='r', alpha=0.5, label='Control', density=True);
plt.hist(ec50_case, 50, color='b', alpha=0.5, label='Case', density=True);
plt.legend(fontsize=20);
plt.xlabel('ec50', fontsize=20);
plt.xticks(fontsize=15);
plt.ylabel('Density', fontsize=20);
plt.yticks(fontsize=15);

Biomarker distributions: ec50 for Cases and Controls.

If we select a different threshold – i.e. instead of 0.5, we can iterate over the range of 0.1 - 0.9, for example, in increments of 0.1 – we generate different biomarkers (ec10, ec20 … ec90). We can treat each biomarker as a different classification model, and assess how powerful that model is at assessing whether someone will develop cardiotoxicity or not. To do so, we’ll create distributions for each biomarker (not shown), and then generate ROC curves and AUC values for each curve.

ROC curves and AUC for each ec-X biomarker level.

This is where my limited domain knowledge comes at a cost – I’m not sure if the biomarkers I’ve chosen (i.e. incremental ec values) are actually biologically relevant. The point, however, is that each biomarker yields a different AUC, which theoretically shows that the Cases and Controls can be differentially distinguished, depending on which biomarker we choose to examine. In this case, ec10 has the most discriminative diagnostic power.

Something I did wonder about while exploring this data was how dependent the ROC curves and AUC statistics are on sample size. Previously, I’d looked at rates of convergence of various estimators – the AUC should also theoretically show some convergence to a “true” value as $n$ increases – but I’m not sure if it follows any sort of relevant distribution. I imagine the AUC is domain-dependent, in that it depends on the distribution of the biomarker of interest? Might be a good idea for another post…

Cheers.

The Delta Method

Tue, 19 Mar 2019 12:43:32 -0700

Here, we’ll look at various applications of the Delta Method, especially in the context of variance stabilizing transformations, along with looking at the confidence intervals of estimates.

The Delta Method is used as a way to approximate the Standard Error of transformations of random variables, and is based on a Taylor Series approximation.

In the univariate case, if we have a random variable, $X_{n}$, that converges in distribution to a $N(0, \sigma^{2})$ distribution, we can apply a function to this random variable as:

$$\begin{align} \sqrt{n}(X_{n} - \theta) \xrightarrow{d} N(0,\sigma^{2}) \\ \sqrt{n}(g(X_{n}) - g(\theta)) \xrightarrow{d} ; ? \end{align}$$

However, we don’t know the asymptotic variance of this transformed variable just yet. In this case, we can approximate our function $g(x)$ using a Taylor Series approximation, evaluated at $\theta$:

$$\begin{align} g(x) = g(\theta) + g\prime(\theta)(x-\theta) + O() \end{align}$$

where $O()$ is the remainder of higher-order Taylor Series terms that converges to 0.

By Slutsky’s Theorem and the Continuous Mapping Theorem, we know that since $\bar{\theta} \xrightarrow{p} \theta$, we know that $g\prime(\bar{\theta}) \xrightarrow{p} g\prime(\theta)$

Plugging this back in to our original equation and applying Slutsky’s Perturbation Theorem, we have:

$$\begin{align} &= \sqrt{n}(\Big[g(\theta) + g\prime(\theta)(x-\theta)\Big] - g(\theta)) \\ &= \sqrt{n}(g\prime(\theta)(x-\theta)) \\ &= g\prime(\theta)\sqrt{n}(X_{n} - \theta) \end{align}$$

and since we know that $\sqrt{n}(\bar{X_{n}} - \theta) \xrightarrow{d} N(0,\sigma^{2})$, we now know that $g\prime(\theta) \sqrt{n}(\bar{X_{n}} - \theta) \xrightarrow{d} N(0,g\prime(\theta)^{2} \sigma^{2})$. As such, we have that:

$$\begin{align} \sqrt{n}(g(X_{n}) - g(\theta)) \xrightarrow{d} N(0, g\prime(\theta)^{2}\sigma^{2}) \end{align}$$

The Delta Method can be generalized to the multivariate case, where, instead of the derivative, we use the gradient vector of our function:

$$\begin{align} \sqrt{n}(g(\bar{X_{n}} - g(\theta)) \xrightarrow{d} N(0, \nabla(g)^{T} \Sigma \nabla(g)) \end{align}$$

Below, I’m going to look at a few examples applying the Delta Method to simple functions of random variables. Then I’ll go into more involved examples applying the Delta Method via Variance Stabilizing Transformations. Oftentimes, the variance of an estimate depends on its mean, which can vary with the sample size. In this case, we’d like to find a function $g(\theta)$, such that, when applied via the Delta Method, the variance is constant as a function of the sample size.

We’ll start by importing the necessary libraries and defining two functions:

%matplotlib inline
import matplotlib.pyplot as plt

from matplotlib import rc
rc('text', usetex=True)

from scipy.stats import norm, poisson, expon
import numpy as np

Here, we define two simple functions – one to compute the difference between our estimate and its population parameter, and the other to compute the function of our random variable as described by the Central Limit Theorem.

def conv_prob(n, est, pop):

 """
 Method to compute the estimate for convergence in probability.
 """

 return (est-pop)

def clt(n, est, pop):

 """
 Method to examine the Central Limit Theorem.
 """

 return np.sqrt(n)*(est-pop)

Let’s have a look at an easy example with the Normal Distribution. We’ll set $\mu = 0$ and $\sigma^{2} = 5$. Remember that when using the Scipy Normal distribution, the norm class accepts the standard deviation, not the variance. We’ll show via the Central Limit Theorem that the function $\sqrt{n}(\bar{X_{n}} - \mu) \xrightarrow{d} N(0,\sigma^{2})$.

# set sample sample sizes, and number of sampling iterations
N = [5,10,50,100,500,1000]
iters = 500

mu = 0; sigma = np.sqrt(5)

# store estimates
norm_clt = {n: [] for n in N}

samples = norm(mu,sigma).rvs(size=(iters,1000))

for n in N:
 for i in np.arange(iters):

 est_norm = np.mean(samples[i,0:n])
 norm_clt[n].append(clt(n, est_norm, mu))

Now let’s plot the results.

# Plot results using violin plots
fig = plt.subplots(figsize=(8,5))

for i,n in enumerate(N):
 temp = norm_clt[n]
 m = np.mean(temp)
 v = np.var(temp)
 print('Sample Size: %i has empirical variance: %.2f' % (n, v.mean()))

 plt.violinplot(norm_clt[n], positions=[i],)

Central Limit Theorem applied to Normal Distribution.

As expected, we see that the Normal distribution mean and variance estimates are independent of the sample size. In this case, we don’t need to apply a variance stabilizing transformation. We also see that the variance fluctuates around $5$. Now, let’s apply a simple function $g(\theta) = \theta^{2}$ to our data. So $g\prime(\theta) = 2\theta$, and the variance of our function becomes $g\prime(\mu)^{2}\sigma^{2} = (2\mu)^{2} \sigma^{2} = 4\mu^{2}\sigma^{2}$. Let’s look at a few plots, as a function of changing $\mu$.

# set sample sample sizes, and number of sampling iterations
mus = [1,2,3,4]

N = [5,10,50,100,500,1000]
iters = 2000
sigma = np.sqrt(5)


fig, ([ax1,ax2], [ax3,ax4]) = plt.subplots(2,2, figsize=(14,9))
for j ,m in enumerate(mus):

 # store estimates
 norm_clt = {n: [] for n in N}
 samples = norm(m, sigma).rvs(size=(iters, 1000))


 plt.subplot(2,2,j+1)
 for k, n in enumerate(N):
 np.random.shuffle(samples)
 for i in np.arange(iters):

 est_norm = np.mean(samples[i, 0:n])
 norm_clt[n].append(clt(n, est_norm**2, m**2))

 plt.violinplot(norm_clt[n], positions=[k],)

Central Limit Theorem applied to function of Normal Distribution.

We see that the variance increases as the mean increases, and that, as the sample sizes increase, the distributions converge to the $$N(0, 4\mu^{2}\sigma^{2})$$ asymptotic distribution.

Variance Stabilization for the Poisson Distribution

Now let’s look at an example where the variance depends on the sample size. We’ll use the Poisson distribution in this case. We know that for the Poisson distribution, the variance is dependent on the mean, so let’s define a random variable, $X_{\lambda}$, where $\lambda = n*\theta$. $n$ is the sample size, and $\theta$ is a fixed constant.

We’ll define $ X_{\lambda } = \sum_{i=1}^{n} X_{\theta}$, the sum of $n$ independent Poisson random variables, so that the expected value and variance of $X_{\lambda } = n\theta$

If we wanted to apply the Central Limit Theorem to $X_{\lambda }$, our convergence would be as follows:

$$\begin{align} \sqrt{n}(X_{\lambda} - \lambda) \xrightarrow{d} N(0,\sigma^{2}(\lambda)) \end{align}$$

where the variance $\sigma^{2}(\lambda)$ depends on the mean, $\lambda$. In order to stabilize the variance of this variable, we can apply the Delta Method, in order to generate a variable that converges to a standard Normal distribution asymptotically.

$$\begin{align} \sqrt{n}(g(X_{\lambda}) - g(\lambda)) \xrightarrow{d} N(0,g\prime(\theta)^{2}\sigma^{2}) \ \end{align}$$

where

$$\begin{align} &g\prime(\theta)^{2} \theta = 1 \\ &g\prime(\theta)^{2} = \frac{1}{\theta} \\ &g\prime(\theta) = \frac{1}{\sqrt{\theta}} \\ &g(\theta) = \int \frac{\partial{\theta}}{\sqrt{\theta}} \\ &g(\theta) = 2\sqrt{\theta} \end{align}$$

is our variance stabilizing function.

def p_lambda(n, theta=0.5):

 """
 Function to compute lambda parameter for Poisson distribution.
 Theta is constant.
 """
 return n*theta

theta = 0.5

N = [5,10,50,100,250,500,750,1000]
iters = 500

clt_pois = {n: [] for n in N}
pois_novar= {n: [] for n in N}
pois_var = {n: [] for n in N}

for n in N:
 for i in np.arange(iters):
 est_mu = np.mean(poisson(mu=(n*theta)).rvs(n))

 pois_novar[n].append(clt(n, est_mu, p_lambda(n)))
 pois_var[n].append(clt(n, 2*np.sqrt(est_mu), 2*np.sqrt(p_lambda(n))))

 clt_pois[n].append(conv_prob(n, est_mu, n*theta))

fig,([ax1, ax2]) = plt.subplots(2,1, figsize=(15,6))

plt.subplot(1,2,1)
for i,n in enumerate(N):
 plt.violinplot(pois_novar[n], positions=[i])

plt.subplot(1,2,2)
for i,n in enumerate(N):
 plt.violinplot(pois_var[n], positions=[i])

Variance stabilization of Poisson distribution.

Variance Stabilization for the Exponential Distribution

Applying the same method to the Exponential distribution, we’ll find that the variance stabilizing transformation is $g(\theta) = log(\theta)$. We’ll apply that here:

theta = 0.5

N = [5,10,50,100,250,500,750,1000]
iters = 500

clt_exp = {n: [] for n in N}
exp_novar= {n: [] for n in N}
exp_var = {n: [] for n in N}

for n in N:
 for i in np.arange(iters):
 samps = expon(scale=n*theta).rvs(n)

 est_mu = np.mean(samps)
 est_var = np.var(samps)

 exp_novar[n].append(clt(n, est_mu, (n*theta)))
 exp_var[n].append(clt(n, np.log(est_mu), np.log(n*theta)))

 clt_exp[n].append(conv_prob(n, est_mu, n*theta))

fig,([ax1, ax2]) = plt.subplots(2,1, figsize=(15,6))

plt.subplot(1,2,1)
for i,n in enumerate(N):
 plt.violinplot(exp_novar[n], positions=[i])

plt.subplot(1,2,2)
for i,n in enumerate(N):
 plt.violinplot(exp_var[n], positions=[i])

Variance stabilization of Exponential distribution.

Example of Standard Error Computation Using Delta Method for Polynomial Regression

As an example of applying the Delta Method to a real-world dataset, I’ve downloaded the banknote dataset from the UCI Machine Learning Repository. In this exercise, I’ll apply the logistic function via logistic regression to assess whether or not a banknote is real or fake, using a set of features. I’ll compute confidence intervals of our prediction probabilities using the Delta Method. There are four unique predictors in this case: the variance, skew, kurtosis, and entropy of the Wavelet-transformed banknote image. I’ll treat each of these predictors independently, using polynomial basis functions of degree 3.

In this example, we’re interested in the standard error of our probability estimate. Our function is the Logistic Function, as follows:

$$\begin{align} g(\beta) &= \frac{1}{1+e^{-x^{T}\beta}} \\ &= \frac{e^{x^{T}\beta}}{1+e^{x^{T}\beta}} \end{align}$$

where the gradient of this multivariate function is:

$$\begin{align} \nabla g(\beta) &= \frac{\partial g}{\partial \beta} e^{x^{T}\beta}(1+e^{x^{T}\beta})^{-1} \\ &= x^{T}e^{x^{T}\beta}(1+e^{x^{T}\beta})^{-1} - x^{T}e^{x^{T}\beta}e^{x^{T}\beta} \\ &= x^{T}\Big(e^{x^{T}\beta}(1+e^{x^{T}\beta})^{-1} - e^{x^{T}\beta}e^{x^{T}\beta}\Big)(1+e^{x^{T}\beta})^{-2} \\ &= x^{T} \frac{e^{x^{T}\beta}}{(1+e^{x^{T}\beta})^{2}} \\ \nabla g(\beta) &= x^{T} g(\beta)(1-g(\beta)) \end{align}$$

so that the final estimate of our confidence interval becomes

$$\begin{align} & \sim N(0,x^{T} g(\beta)(1-g(\beta)) \Sigma g(\beta)(1-g(\beta))x) \\ & \sim N(0, \nabla g(\beta)^{T} \Sigma \nabla g(\beta)) \end{align}$$

import pandas as pd
import statsmodels.api as sm
from sklearn.preprocessing import PolynomialFeatures

import pandas as pd
import statsmodels.api as sm
from sklearn.preprocessing import PolynomialFeatures

bank = pd.read_csv('/Users/kristianeschenburg/Documents/Statistics/BankNote.txt',
 sep=',',header=None, names=['variance', 'skew', 'kurtosis', 'entropy','class'])
bank.head()

fig = plt.subplots(2,2, figsize=(12,8))
for j, measure in enumerate(['variance', 'kurtosis', 'skew', 'entropy']):

 predictor = np.asarray(bank[measure])
 response = np.asarray(bank['class'])

 idx = (response == 1)

 # plot test set
 plt.subplot(2,2,j+1)
 plt.violinplot(predictor[idx], positions=[1]);
 plt.violinplot(predictor[~idx], positions=[0])
 plt.title('{:} By Classification'.format(measure), fontsize=18)
 plt.ylabel('Measure: {:}'.format(measure),fontsize=15)
 plt.yticks(fontsize=13)
 plt.xticks([0,1],['Fake','Real'], fontsize=15)

plt.tight_layout()

Bank note feature distributions, based on note class.

Based on the above plot, we can see that variance, skew, and kurtosis seem to be the most informative, while the entropy distributions do not seem to be that different based on bank note class.

Next, we fit a logistic regression model of note classification on note feature, with polynomial order of degree 3. We then compute the standard errors of the transformed variance. It was transformed using the logistic function, so we’ll need to compute the gradient of this function.

fig = plt.subplots(2,2, figsize=(12,8))
for j, measure in enumerate(['variance', 'kurtosis', 'skew', 'entropy']):

 # Generate polynomial object to degree 
 # transform age to 4-degree basis function
 poly = PolynomialFeatures(degree=2)
 idx_order = np.argsort(bank[measure])

 predictor = bank[measure][idx_order]
 response = bank['class'][idx_order]

 features = poly.fit_transform(predictor.values.reshape(-1,1));

 # fit logit curve to curve
 logit = sm.Logit(response, features).fit();

 test_features = np.linspace(np.min(predictor), np.max(predictor), 100)
 test_features = poly.fit_transform(test_features.reshape(-1,1))
 # predict on test set
 class_prob = logit.predict(test_features)

 cov = logit.cov_params()
 yx = (class_prob*(1-class_prob))[:,None] * test_features
 se = np.sqrt(np.diag(np.dot(np.dot(yx, cov), yx.T)))

 # probability can't exceed 1, or be less than 0
 upper = np.maximum(0, np.minimum(1, class_prob+1.96*se))
 lower = np.maximum(0, np.minimum(1, class_prob-1.96*se))

 # plot test set
 plt.subplot(2,2,j+1)
 plt.plot(test_features[:, 1], class_prob);
 plt.plot(test_features[:, 1], upper, color='red', linestyle='--', alpha=0.5);
 plt.plot(test_features[:, 1], lower, color='red', linestyle='--', alpha=0.5);
 plt.title(r'P(isReal \Big| X)', fontsize=18)
 plt.xlabel('{:}'.format(measure),fontsize=15)
 plt.ylabel('Probability',fontsize=15)
 plt.grid(True)

plt.tight_layout()

Confidence intervals for each feature, computed using Delta Method.

Mahalanobis Distances of Brain Connectivity

Fri, 07 Dec 2018 05:12:32 -0700

For one of the projects I’m working on, I have an array of multivariate data relating to brain connectivity patterns. Briefly, each brain is represented as a surface mesh, which we represent as a graph $G = (V,E)$, where $V$ is a set of $n$ vertices, and $E$ is the set of edges between vertices.

Additionally, for each vertex $v \in V$, we also have an associated scalar label, which we’ll denote $l(v)$, that identifies what region of the cortex each vertex belongs to, the set of regions which we define as $L = {1, 2, … k}$. And finally, for each vertex $v \in V$, we also have a multivariate feature vector $r(v) \in \mathbb{R}^{1 \times k}$, that describes the strength of connectivity between it, and every region $l \in L$.

Example of cortical map, and array of connectivity features.

I’m interested in examining how “close” the connectivity samples of one region, $l_{j}$, are to another region, $l_{k}$. In the univariate case, one way to compare a scalar sample to a distribution is to use the $t$-statistic, which measures how many standard deviations away from the mean a given sample is:

$$\begin{align} t_{s} = \frac{\bar{x} - \mu}{\frac{s}{\sqrt{n}}} \end{align}$$

where $\mu$ is the population mean, and $s$ is the sample standard deviation. If we square this, we get:

$$\begin{align} t^{2} = \frac{(\bar{x} - \mu)^{2}}{\frac{s^{2}}{n}} = \frac{n (\bar{x} - \mu)^{2}}{S^{2}} \sim F(1,n) \end{align}$$

We know the last part is true, because the numerator and denominator are independent $\chi^{2}$ distributed random variables. However, I’m not working with univariate data – I have multivariate data. The multivariate generalization of the $t$-statistic is the Mahalanobis Distance:

$$\begin{align} d &= \sqrt{(\bar{x} - \mu)\Sigma^{-1}(\bar{x}-\mu)^{T}} \end{align}$$

where the squared Mahalanobis Distance is:

$$\begin{align} d^{2} &= (\bar{x} - \mu)\Sigma^{-1}(\bar{x}-\mu)^{T} \end{align}$$

where $\Sigma^{-1}$ is the inverse covariance matrix. If our $X$’s were initially distributed with a multivariate normal distribution, $N_{p}(\mu,\Sigma)$ (assuming $\Sigma$ is non-degenerate i.e. positive definite), the squared Mahalanobis distance, $d^{2}$ has a $\chi^{2}_{p}$ distribution. We show this below.

We know that $(X-\mu)$ is distributed $N_{p}(0,\Sigma)$. We also know that, since $\Sigma$ is symmetric and real, that we can compute the eigendecomposition of $\Sigma$ as:

$$\begin{align} \Sigma = U \Lambda U^{T} \end{align}$$

and consequently, because $U$ is an orthogonal matrix, and because $\Lambda$ is diagonal, we know that $\Sigma^{-1}$ is:

$$\begin{align} \Sigma^{-1} &= (U \Lambda U^{T})^{-1} \\ &= U \Lambda^{-1} U^{T} \\ &= (U \Lambda^{\frac{-1}{2}}) (U \Lambda^{\frac{-1}{2}})^{T} \\ &= R R^{T} \end{align}$$

Therefore, we know that $R^{T}(X-\mu) \sim N_{p}(0,I_{p})$:

$$\begin{align} X &\sim N_{p}(\mu,\Sigma) \\ (X-\mu) = Y &\sim N_{p}(0,\Sigma)\\ R^{T}Y = Z &\sim N_{p}(0, R^{T} \Sigma R) \\ &\sim N_{p}(0, \Lambda^{\frac{-1}{2}} U^{T} (U \Lambda U^{T}) U \Lambda^{\frac{-1}{2}}) \\ &\sim N_{p}(0, \Lambda^{\frac{-1}{2}} I_{p} \Lambda I_{p} \Lambda^{\frac{-1}{2}}) \\ &\sim N_{p}(0,I_{p}) \end{align}$$

so that we have

$$\begin{align} &= (X-\mu)\Sigma^{-1}(X-\mu)^{T} \\ &= (X-\mu)RR^{T}(X-\mu)^{T} \\ &= Z^{T}Z \end{align}$$

the sum of $p$ squared standard Normal random variables, which is the definition of a $\chi_{p}^{2}$ distribution with $p$ degrees of freedom. So, given that we start with a $MVN$ random variable, the squared Mahalanobis distance is $\chi^{2}_{p}$ distributed. Because the sample mean and sample covariance are consistent estimators of the population mean and population covariance parameters, we can use these estimates in our computation of the Mahalanobis distance.

Also, of particular importance is the fact that the Mahalanobis distance is not symmetric. That is to say, if we define the Mahalanobis distance as:

$$\begin{align} M(A, B) = \sqrt{(A - \mu(B))\Sigma(B)^{-1}(A-\mu(B))^{T}} \end{align}$$

then $M(A,B) \neq M(B,A)$, clearly. Because the parameter estimates are not guaranteed to be the same, it’s straightforward to see why this is the case.

Now, back to the task at hand. For a specified target region, $l_{T}$, with a set of vertices, $V_{T} = {v \; : \; l(v) \; = \; l_{T}, \; \forall \; v \in V}$, each with their own distinct connectivity fingerprints, I want to explore which areas of the cortex have connectivity fingerprints that are different from or similar to $l_{T}$’s features, in distribution. I can do this by using the Mahalanobis Distance. And based on the analysis I showed above, we know that the data-generating process of these distances is related to the $\chi_{p}^{2}$ distribution.

First, I’ll estimate the covariance matrix, $\Sigma_{T}$, of our target region, $l_{T}$, using the Ledoit-Wolf estimator (the shrunken covariance estimate has been shown to be a more reliable estimate of the population covariance), and mean connectivity fingerprint, $\mu_{T}$. Then, I’ll compute $d^{2} = M^{2}(A,A)$ for every $\{v: v \in V_{T}\}$. The empirical distribution of these distances should follow a $\chi_{p}^{2}$ distribution. If we wanted to do hypothesis testing, we would use this distribution as our null distribution.

Next, in order to assess whether this intra-regional similarity is actually informative, I’ll also compute the similarity of $l_{T}$ to every other region, $\{ l_{k} \; : \; \forall \; k \in L \setminus \{T\} \}$ – that is, I’ll compute $M^{2}(A, B) \; \forall \; B \in L \setminus T$. If the connectivity samples of our region of interest are as similar to one another as they are to other regions, then $d^{2}$ doesn’t really offer us any discriminating information – I don’t expect this to be the case, but we need to verify this.

Then, as a confirmation step to ensure that our empirical data actually follows the theoretical $\chi_{p}^{2}$ distribution, I’ll compute the location and scale Maximum Likelihood(MLE) parameter estimates of our $d^{2}$ distribution, keeping the d.o.f. (i.e. $p$) fixed.

See below for Python code and figures…

Step 1: Compute Parameter Estimates

%matplotlib inline
import matplotlib.pyplot as plt

from matplotlib import rc
rc('text', usetex=True)

import numpy as np
from scipy.spatial.distance import cdist
from scipy.stats import chi2, probplot

from sklearn import covariance

# lab_map is a dictionary, mapping label values to sample indices
# our region of interest has a label of 8
LT = 8

# get indices for region LT, and rest of brain
lt_indices = lab_map[LT]
rb_indices = np.concatenate([lab_map[k] for k in lab_map.keys() if k != LT])

data_lt = conn[lt_indices, :]
data_rb = conn[rb_indices, :]

# fit covariance and precision matrices
# Shrinkage factor = 0.2
cov_lt = covariance.ShrunkCovariance(assume_centered=False, shrinkage=0.2)
cov_lt.fit(data_lt)
P = cov_lt.precision_

Next, compute the Mahalanobis Distances:

# LT to LT Mahalanobis Distance
dist_lt = cdist(data_lt, data_lt.mean(0)[None,:], metric='mahalanobis', VI=P)
dist_lt2 = dist_lt**2

# fit covariance estimate for every region in cortical map
EVs = {l: covariance.ShrunkCovariance(assume_centered=False,
 shrinkage=0.2) for l in labels}

for l in lab_map.keys():
 EVs[l].fit(conn[lab_map[l],:])

# compute d^2 from LT to every cortical region
# save distances in dictionary
lt_to_brain = {}.fromkeys(labels)
for l in lab_map.keys():

 temp_data = conn[label_map[l], :]
 temp_mu = temp_data.mean(0)[None, :]

 temp_mh = cdist(data_lt, temp_mu, metric='mahalanobis', VI=EVs[l].precision_)
 temp_mh2 = temp_mh**2

 lt_to_brain[l] = temp_mh2

# plot distributions seperate (scales differ)
fig = plt.subplots(2,1, figsize=(12,12))
plt.subplot(2,1,1)
plt.hist(lt_to_brain[LT], 50, density=True, color='blue',
 label='Region-to-Self', alpha=0.7)

plt.subplot(2,1,2)
for l in labels:
 if l != LT:
 plt.hist(lt_to_brain[l], 50, density=True, linewidth=2,
 alpha=0.4, histtype='step')

Empirical distributions of within-region (top) and between-region (bottom) $d^{2}$ values. Each line is the distribution of the distance of samples in our ROI to a whole region.

As expected, the distribution of $d^{2}$, the distance of samples in our region of interest, $l_{T}$, to distributions computed from other regions, is (considerably) larger and much more variable, while the profile of points within $l_{T}$ looks to have much smaller variance – this is good! This means that we have high intra-regional similarity when compared to inter-regional similarities. This fits what’s known in neuroscience as the “cortical field hypothesis”.

Step 2: Distributional QC-Check

Because we know that our data should follow a $\chi^{2}_{p}$ distribution, we can fit the MLE estimate of our location and scale parameters, while keeping the $df$ parameter fixed.

p = data_lt.shape[1]
mle_chi2_theory = chi2.fit(dist_lt2, fdf=p)

xr = np.linspace(data_lt.min(), data_lt.max(), 1000)
pdf_chi2_theory(xr, *mle_chi2_theory)

fig = plt.subplot(1,2,2,figsize=(18, 6))

# plot theoretical vs empirical null distributon
plt.subplot(1,2,1)
plt.hist(data_lt, density=True, color='blue', alpha=0.6,
 label = 'Empirical')
plt.plot(xr, pdf_chi2_theory, color='red',
 label = '$\chi^{2}_{p}')

# plot QQ plot of empirical distribution
plt.subplot(1,2,2)
probplot(D2.squeeze(), sparams=mle_chi2_theory, dist=chi2, plot=plt);

Density and QQ plot of null distribution.

From looking at the QQ plot, we see that the empirical density fits the theoretical density pretty well, but there is some evidence that the empirical density has heavier tails. The heavier tail of the upper quantile could probability be explained by acknowledging that our starting cortical map is not perfect (in fact there is no “gold-standard” cortical map). Cortical regions do not have discrete cutoffs, although there are reasonably steep gradients in connectivity. If we were to include samples that were considerably far away from the the rest of the samples, this would result in inflated densities of higher $d^{2}$ values.

Likewise, we also made the distributional assumption that our connectivity vectors were multivariate normal – this might not be true – in which case our assumption that $d^{2}$ follows a $\chi^{2}_{p}$ would also not hold.

Finally, let’s have a look at some brains! Below, is the region we used as our target – the connectivity profiles from vertices in this region were used to compute our mean vector and covariance matrix – we compared the rest of the brain to this region.

Region of interest.

Estimated squared Mahalanobis distances, overlaid on cortical surface.

Here, larger $d^{2}$ values are in red, and smaller $d^{2}$ are in black. Interestingly, we do see pretty large variance of $d^{2}$ spread across the cortex – however the values are smoothly varying, but there do exists sharp boundaries. We kind of expected this – some regions, though geodesically far away, should have similar connectivity profiles if they’re connected to the same regions of the cortex. However, the regions with connectivity profiles most different than our target region are not only contiguous (they’re not noisy), but follow known anatomical boundaries, as shown by the overlaid boundary map.

This is interesting stuff – I’d originally intended on just learning more about the Mahalanobis Distance as a measure, and exploring its distributional properties – but now that I see these results, I think it’s definitely worth exploring further!

Convergence In Probability

Wed, 28 Nov 2018 13:12:32 -0700

I’m going over Chapter 5 in Casella and Berger’s (CB) “Statistical Inference”, specifically Section 5.5: Convergence Concepts, and wanted to document the topic of convergence in probability with some plots demonstrating the concept.

From CB, we have the definition of convergence in probability: a sequence of random variables $X_{1}, X_{2}, … X_{n}$ converges in probability to a random variable $X$, if for every $\epsilon > 0$,

$$\begin{align} \lim_{n \to \infty} P(| X_{n} - X | \geq \epsilon) = 0 \\ \end{align}$$

Intuitively, this means that, if we have some random variable $X_{k}$ and another random variable $X$, the absolute difference between $X_{k}$ and $X$ gets smaller and smaller as $k$ increases. The probability that this difference exceeds some value, $\epsilon$, shrinks to zero as $k$ tends towards infinity. Using convergence in probability, we can derive the Weak Law of Large Numbers (WLLN):

$$\begin{align} \lim_{n \to \infty} P(|\bar{X}_{n} - \mu | \geq \epsilon) = 0 \end{align}$$

which we can take to mean that the sample mean converges in probability to the population mean as the sample size goes to infinity. If we have finite variance (that is $Var(X) < \infty$), we can prove this using Chebyshev’s Inequality

$$\begin{align} &= P(|\bar{X}_{n} - \mu | \geq \epsilon) \\ &= P((\bar{X}_{n} - \mu)^{2} \geq \epsilon^{2}) \leq \frac{E\Big[(\bar{X}_{n} - \mu)^{2}\Big]}{\epsilon^{2}} \\ &= P((\bar{X}_{n} - \mu)^{2} \geq \epsilon^{2}) \leq \frac{Var(\bar{X_{n}})}{\epsilon^{2}} \\ &= P((\bar{X}_{n} - \mu)^{2} \geq \epsilon^{2}) \leq \frac{\sigma^{2}}{n^{2}\epsilon^{2}} \end{align}$$

where $\frac{\sigma^{2}}{n^{2} \epsilon^{2}} \rightarrow 0$ as $n \rightarrow \infty$. Intuitively, this means, that the sample mean converges to the population mean – and the probability that their difference is larger than some value is bounded by the variance of the estimator. Because we showed that the variance of the estimator (right hand side) shrinks to zero, we can show that the difference between the sample mean and population mean converges to zero.

We can also show a similar WLLN result for the sample variance using Chebyshev’s Inequality, as:

$$\begin{align} S_{n}^{2} = \frac{1}{n-1} \sum_{i=1}^{n} (X_{i} - \bar{X}_{n})^{2} \end{align}$$

using the unbiased estimator, $S_{n}^{2}$, of $\sigma^{2}$ as follows:

$$\begin{align} P(|S_{n}^{2} - \sigma^{2}| \geq \epsilon) \leq \frac{E\Big[(S_{n}^{2} - \sigma^{2})^{2}\Big]}{\epsilon^{2}} = \frac{Var(S_{n}^{2})}{\epsilon^{2}} \end{align}$$

so all we need to do is show that $Var(S_{n}^{2}) \rightarrow 0$ as $n \rightarrow \infty$.

Let’s have a look at some (simple) real-world examples. We’ll start by sampling from a $N(0,1)$ distribution, and compute the sample mean and variance using their unbiased estimators.

# Import numpy and scipy libraries
import numpy as np
from scipy.stats import norm

%matplotlib inline
import matplotlib.pyplot as plt

plt.rc('text', usetex=True)

# Generate set of samples sizes
samples = np.concatenate([np.arange(0, 105, 5),
 10*np.arange(10, 110, 10),
 100*np.arange(10, 210, 10)])

# number of repeated samplings for each sample size
iterations = 500

# store sample mean and variance
means = np.zeros((iterations, len(samples)))
vsrs = np.zeros((iterations, len(samples)))

for i in np.arange(iterations):
 for j, s in enumerate(samples):

 # generate samples from N(0,1) distribution
 N = norm.rvs(loc=0, scale=1, size=s)
 mu = np.mean(N)

 # unbiased estimate of variance
 vr = ((N - mu)**2).sum()/(s-1)

 means[i, j] = mu
 vsrs[i, j] = vr

Let’s have a look at the sample means and variances as a function of the sample size. Empirically, we see that both the sample mean and variance estimates converge to their population parameters, 0 and 1.

Sample mean estimates as a function of sample size.

Sample variance estimates as a function of sample size.

Below is a simple method to compute the empirical probability that an estimate exceeds the epsilon threshold.

def ecdf(data, pparam, epsilon):

 """
 Compute empirical probability P( |estimate - pop-param| < epsilon).

 Parameters:
 - - - - -
 data: array, float
 array of samples
 pparam: float
 true population parameter
 epsilon: float
 threshold value
 """

 compare = (np.abs(data - pparam) < epsilon)
 prob = compare.mean(0)

 return prob

# test multiple epsilon thresholds
e = [0.9, 0.75, 0.5, 0.25, 0.1, 0.05, 0.01]

mean_probs = []
vrs_probs = []
# compute empirical probabilities at each threshold
for E in e:
 mean_probs.append(1 - ecdf(means, pparam=0, epsilon=E))
 vrs_probs.append(1-ecdf(vsrs, pparam=1, epsilon=E))

Empirical probability that mean estimate exceeds population mean by epsilon.

Empirical probability that variance estimate exceeds population variance by epsilon.

The above plots show that, as sample size increases, the mean estimator and variance estimator both converge to their true population parameters. Likewise, examining the empirical probability plots, we can see that the probability that either estimate exceeds the epsilon thresholds shrinks to zero as the sample size increases.

If we wish to consider a stronger degree of convergence, we can consider convergence almost surely, which says the following:

$$\begin{align} P(\lim_{n \to \infty} |X_{n} - X| \geq \epsilon) = 0 \ \end{align}$$

which considers the entire joint distribution of estimates $( X_{1}, X_{2}…X_{n}, X)$, rather than all pairwise estimates $(X_{1},X), (X_{2},X)… (X_{n},X)$ – the entire set of estimates must converge to $X$ as the sample size approaches infinity.

Overview of Poisson-Multinomial Relationship

Thu, 08 Nov 2018 01:12:32 -0700

In this post, I’m going to briefly cover the relationship between the Poisson distribution and the Multinomial distribution.

Let’s say that we have a set of independent, Poisson-distributed random variables $Y_{1}, Y_{2}… Y_{k}$ with rate parameters $\lambda_{1}, \lambda_{2}, …\lambda_{k}$. We can model the sum of these random variables as a new random variable $N = \sum_{i=1}^{k} Y_{i}$.

Let start with $k=2$. We can define the distrbution of $F_{N}(n)$ as follows:

$$\begin{align} &= P(N \leq n) \\ &= P(Y_{1} + Y_{2} \leq n) \\ &= P(Y_{1} = y_{1}, Y_{2} = n - y_{1}) \\ &= P(Y_{1} = y_{1}) \cdot P(Y_{2} = n-y_{1}) \\ &= \sum_{y_{1}=0}^{n} \frac{e^{-\lambda_{1}}\lambda_{1}^{y_{1}}}{y_{1}!} \cdot \frac{e^{-\lambda_{2}}\lambda_{2}^{n-y_{1}}}{(n-y_{1})!} \\ &= e^{-(\lambda_{1}+\lambda_{2})} \sum_{y_{1}=0}^{n} \frac{\lambda_{1}^{y_{1}}\lambda_{2}^{n-y_{1}}}{y_{1}!(n-y_{1})!} \\ &= e^{-(\lambda_{1}+\lambda_{2})} \sum_{y_{1}=0}^{n} \frac{n!}{n!}\frac{\lambda_{1}^{y_{1}}\lambda_{2}^{n-y_{1}}}{y_{1}!(n-y_{1})!} \\ &= \frac{e^{-(\lambda_{1}+\lambda_{2})}}{n!} \sum_{y_{1}=0}^{n} {n\choose y_{1}} \lambda_{1}^{y_{1}}\lambda_{2}^{n-y_{1}} \end{align}$$

Here, we can apply the Binomial Theorem to the summation to get the following (remember that the Binomial Theorem says, for two numbers $x$ and $y$, that $(x+y)^{n} = \sum_{i=0}^{n} {n \choose i}x^{i}y^{n-i}$):

$$\begin{align} \frac{e^{-(\lambda_{1}+\lambda_{2})}(\lambda_{1} + \lambda_{2})^{n}}{n!} \\ \end{align}$$

which we see is in fact just another Poisson distribution with rate parameter equal to $\lambda_{1} + \lambda_{2}$. This shows that the sum of independent Poisson distributed random variables is also a Poisson random variable, with rate parameter equal to the sum of the univariate rates. By induction, we see that for $k$ independent Poisson distributed random variables $Y_{1}…Y_{k}$, their sum $\sum_{i=1}^{k} Y_{i} \sim Poisson(\sum_{i=1}^{k} \lambda_{i})$.

Now let’s say we’re interested in modeling the conditional distribution of $(Y_{1}…Y_{k}) \mid \sum_{i=1}^{k} = n$. By definition of conditional probability, we have that

$$\begin{align} P(\bar{Y} \mid N=n) &= \frac{P(\bar{Y} ; \cap ; N=n)}{P(N=n)} \\ &= \frac{P(\bar{Y})}{P(N=n)} \end{align}$$

We have the following:

$$\begin{align} P(\bar{Y} \mid N=n) &= \frac{P(\bar{Y} \; \cap \; N=n)}{P(N=n)} \\ &= \Big( \prod_{i=1}^{k} \frac{e^{-\lambda_{i}} \cdot \lambda_{i}^{y_{i}}}{y_{i}!} \Big) \Big/ \frac{e^{-\sum_{i=1}^{k} \lambda_{i}}(\sum_{i}^{k} \lambda_{i})^{n}}{n!} \\ &= \Big( \frac{ e^{-\sum_{i=1}^{k}} \prod_{i=1}^{k} \lambda_{i}^{y_{i}}}{\prod_{i=1}^{k} y_{i}!} \Big) \Big/ \frac{e^{-\sum_{i=1}^{k} \lambda_{i}}(\sum_{i}^{k} \lambda_{i})^{n}}{n!} \\ &= { n \choose y_{1}, y_{2}, …y_{k}} \frac{\prod_{i=1}^{k} \lambda_{i}^{y_{i}}} { \sum_{i}^{k} \lambda_{i})^{n}} \\ &= { n \choose y_{1}, y_{2}, …y_{k}} \prod_{i=1}^{k} \Big( \frac{ \lambda_{i} }{\sum_{i}^{k} \lambda_{i}} \Big)^{y_{i}} \\ &\sim MultiNom(n; \frac{\lambda_{1}}{\sum_{i=1}^{k}}, \frac{\lambda_{2}}{\sum_{i=1}^{k}}, … \frac{\lambda_{k}}{\sum_{i=1}^{k}}) \end{align}$$

So finally, we see that, given the sum of independent Poisson random variables, that conditional distribution of each element of the Poisson vector is Multinomial distributed, with count probabilities scaled by the sum of the individual rates. Importantly, we can extend these ideas (specifically the sum of independent Poisson random variables) to other models, such as splitting and merging homogenous and non-homogenous Poisson Point Processes.

Exploring Neurological Dynamical Systems: Part 2

Thu, 24 May 2018 11:30:43 -0700

In my previous post on dynamic mode decomposition, I discussed the foundations of DMD as a means for linearizing a dynamical system¹²³. In this post, I want to look at a way in which we can use rank-updates to incorporate new information into the spectral decomposition of our linear operator, $A$, in the event that we are generating online measurements from our dynamical system⁴ – see the citation below if you want a more-detailed overview of this topic along with open source code for testing this method.

Recall that we are given an initial data matrix

$$\begin{align} X = \begin{bmatrix} x_{n_{1},m_{1}} & x_{n_{1},m_{2}} & x_{n_{1},m_{3}} & … \\ x_{n_{2},m_{1}} & x_{n_{1},m_{2}} & x_{n_{2},m_{3}} & … \\ x_{n_{3},m_{1}} & x_{n_{1},m_{2}} & x_{n_{3},m_{3}} & … \\ … & … & … & … \\ \end{bmatrix} \in R^{n \times m} \end{align}$$

which we can split into two matrices, shifted one unit in time apart:

$$\begin{align} X^{\ast} &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_1 & \vec{x}_2 & \dots & \vec{x}_{m-1} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \\ Y &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_2 & \vec{x}_3 & \dots & \vec{x}_{m} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \end{align}$$

and we are interested in solving for the linear operator $A$, such that

$$\begin{align} Y = AX^{\ast} \end{align}$$

For simplicity, since we are no longer using the full matrix, I’ll just refer to $X^{\ast}$ as $X$. In the previous post, we made the constraint that $n > m$, and that rank($X$) $\leq m < n$. Here, however, we’ll reverse this assumption, such that $m > n$, and that rank($X$) $\leq m < n$, such that $XX^{T}$ is invertible, so by multiplying both sides by $X^{T}$ we have

$$\begin{align} AXX^{T} &= YX^{T} \\ A &= YX^{T}(XX^{T})^{-1} \\ A &= QP_{x} \end{align}$$

where $Q = YX^{T}$ and $P_{x} = (XX^{T})^{-1}$. Now, let’s say you observe some new data $x_{m+1}, y_{m+1}$, and you want to incorporate this new data into your $A$ matrix. As in the previous post on rank-one updates, we saw that directly computing the inverse could potentially be costly, so we want to refrain from doing that if possible. Instead, we’ll use the Sherman-Morrison-Woodbury theorem again to incorporate our new $x_{m+1}$ sample into our inverse matrix, just as before:

$$\begin{align} (X_{m+1}X^{T}{m+1})^{-1} = P{x} + \frac{P_{x}x_{m+1}x_{m+1}^{T}P_{x}}{1 + x_{m+1}^{T}P_{x}x_{m+1}} \end{align}$$

Likewise, since we’re appending new data to our $Y$ and $X$ matrices, we also have

$$\begin{align} Y_{m+1} = \begin{bmatrix} Y & y_{m+1} \end{bmatrix} \\ X_{m+1} = \begin{bmatrix} \\ X & x_{m+1} \end{bmatrix} \\ \end{align}$$

such that

$$\begin{align} Y_{m+1} X_{m+1}^{T} &= YX^{T} + y_{m+1}x_{m+1}^{T} \\ &= Q + y_{m+1}x_{m+1}^{T} \end{align}$$

which is simply the sum of our original matrix $Q$, plus a rank-one matrix. The authors go on to describe some pretty cool “local” DMD schemes, by incorporating weights, as well as binary thresholds, that are time-dependent into the computation of the linear operator, $A$.

P.J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics 656.1. 2010. ↩︎
Tu et al. On Dynamic Mode Decomposition: Theory And Applications ↩︎
Kunert-Graf et al. Extracting Reproducible Time-Resolved Resting State Networks Using Dynamic Mode Decomposition. Front. Comput. Neurosci. 2019. ↩︎
Zhang et al. Online dynamic mode decomposition for time-varying systems. 2017. ↩︎

Exploring Neurological Dynamical Systems: Part 1

Tue, 22 May 2018 14:02:52 -0700

In the next two posts, I want to talk briefly about an algorithm called Dynamic Mode Decomposition (DMD). DMD is a spatiotemporal modal decomposition technique that can be used to identify spatial patterns in a signal (modes), along with the time course of these spatial patterns (dynamics). As such, the algorithm assumes that the input data has both a spatial and a temporal component. We are interested in modeling how the system evolves over time.

If you’d like to find more information about DMD, Peter Schmid¹ and Jonathan Tu² have written excellent expositions on the topic. Likewise, if you’d like to follow along with the code for the following analysis, see my repo. For a more in-depth analysis that applies DMD to brain activity in the resting brain, see a recent publication by my colleagues and me³, along with the code used for our analysis.

The DMD Algorithm

Let’s assume that you’ve taken $n$ measurements from specific points in space for $m$ time points, where for now we assume that $m\lt n$. For now, we’ll assume that the sampling frequency, $\omega$, is stable across the entire experiment. We define our entire data matrix as

We are interested in solving for the matrix, $A \in R^{n \times n}$, such that

$$\begin{align} x_{t+1} = A x_{t} \; \; \forall \; \; t = 1,2,…m-1 \end{align}$$

Given our full data matrix $X$, we can define two matrices $X^{*}$ and $Y$ such that

$$\begin{align} X^{*} &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_{1} & \vec{x}_{2} & \dots & \vec{x}_{m-1} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \\ Y &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_2 & \vec{x}_3 & \dots & \vec{x}_{m} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \end{align}$$

so that we can write

$$\begin{align} Y = AX^{\ast} \end{align}$$

If $n$ is small, this is relatively easy to compute – however, if $n$ is large, as is the case when modeling temporal dynamics in resting-state MRI, it would be computationally inefficient to compute A directly. To alleviate this, we can make use of the Singular Value Decomposition (SVD) of our predictor matrix $X^{\ast}$. We define the SVD of $X^{\ast}$ as

$$\begin{align} X^{\ast} = U \Sigma V^{T} \ \end{align}$$

as well as the Moore-Penrose pseudo-inverse of $X^{\ast} = X^{\dagger}$ as

$$\begin{align} X^{\dagger} = V \Sigma^{-1} U^{T} \\ \end{align}$$

such that we can write

$$\begin{align} YX^{\dagger} = YV \Sigma^{-1} U^{T} = A X^{\ast}X^{\dagger} = A \\ \end{align}$$

Additionally, if we assume that $rank(X^{\ast}) = r \leq m$, then we can use the truncated SVD such that

$$\begin{align} U & \in R^{n \times r} \\ V^{T} & \in R^{r \times m} \\ \end{align}$$

and

$$\begin{align} \Sigma = \begin{bmatrix} \sigma_{1} & 0 & 0 & … \\ 0 & \sigma_{2} & 0 & … \\ 0 & 0 & \ddots & … \\ \vdots & \vdots & \vdots & \sigma_{r} \\ \end{bmatrix} \in R^{r \times r} \end{align}$$

As it stands now, we still compute an $A \in R^{n \times n}$ matrix. However, because we have a potentially low-rank system, we can apply a Similarity Transformation to $A$ in order to reduce its dimensionality, without changing its spectrum. Using our spatial singular vectors $U$, we define

$$\begin{align} \tilde{A} &= U^{T} A U \\ &= U^{T} (YV \Sigma^{-1} U^{T}) U \\ &= U^{T} Y V \Sigma^{-1} \\ \end{align}$$

where $\tilde{A} \in R^{r \times r}$. If we consider the above SVD, we see that $U$ is the matrix of left singular vectors, an orthogonal basis that spans $C(X^{\ast})$, which is an r-dimensional subspace of $R^{n}$. Thus, the similarity transform represents a mapping $f(A) = U^{T} A U : R^{n} \rightarrow R^{r}$. We now have a reduced-dimensional representation of our linear operator, from which we can compute the spatial modes and dynamic behavior of each mode. First, however, because of the notion of variance captured by the singular values of our original predictor matrix, we weight $\tilde{A}$ by the singular values as

$$\begin{align} \hat{A} = \Sigma^{\frac{1}{2}} \tilde{A} \Sigma^{\frac{1}{2}} \\ \end{align}$$

such that our computed spatial modes have been weighted by the amount they contribute to our measured signal. We can now compute the eigendecomposition of $\hat{A}$ as

$$\begin{align} \hat{A} W = W \Lambda \\ \end{align}$$

where the eigenvectors $W$ are the reduced-dimension representations of our spatial modes, and the eigenvalues $\Lambda$ capture the dynamic behavior of our spatial modes. Because our original data matrix $X^{\ast}$ had spatial dimension $n$ and our eigenvectors have dimension $r$, we need to up-project our eigenvectors $W$ to compute the final spatial modes, via

$$\begin{align} \Phi = Y V \Sigma^{\frac{-1}{2}}W \end{align}$$

From the SVD of our prediction matrix $X^\ast=U \Sigma V^{T}$, the matrix $V \in R^{m \times r}$ is the matrix of right singular vectors, an orthogonal basis spanning the space of $X^{\ast T}$ (i.e. $r$ basis vectors spanning the space of the measured time courses). Thus, we see that $H = (V \Sigma^{\frac{-1}{2}})W$ represents a linear combination of the temporal basis vectors (a mapping from $R^{r} \rightarrow R^{m}$) for each eigenvector $w_{i}$ of $W$, weighted by the corresponding singular value $\sigma_{i}^{\frac{-1}{2}}$ (that acts to normalize the spatial mode amplitudes). Finally, we see that $\Phi = X^{\ast}H$ computes how much of each temporal basis vector is present in the measured time course at each point in space.

Because we are modeling a dynamical system, we can compute the continuous time dynamics of our system using our spatial modes and eigenvalues as

$$\begin{align} \vec{x}(t) \approx \sum_{i=1}^{r} b_{i}\exp^{((\gamma_{i} + 2i\pi f_{i})\cdot t)} \vec{\phi}_{i} \end{align}$$

where $\gamma_{i}$ is a growth-decay constant and $f_{i}$ is the frequency of oscillation of the spatial mode $\phi_{i}$. We can compute these two constants as

$$\begin{align} \gamma_{i} &= \frac{\text{real}(\text{ln}(\lambda_{i}))}{\Delta t} \\ f_{i} &= \frac{\text{imag}(\text{ln}(\lambda_{i}))}{2\pi \Delta t} \end{align}$$

So, we can see that DMD linearizes our measured time series, by fitting what can be analogized to a “global” regression. That is, instead of computing how a single time point predicts the next time point, which could readily be solved using the simple Normal equations, DMD computes how a matrix of time points predicts another matrix of time points that is shifted one unit of time into the future. To this extent, DMD minimizes the Frobenius norm of

$$\begin{align} \min \limits_{A} \lVert Y - AX^{\ast} \rVert^{2}_{F} \\ \end{align}$$

However, rather than explicitly computing the matrix $A$, DMD computes the eigenvectors and eigenvalues of $A$, by utilizing the Singular Value Decomposition, along with a Similarity Transformation, in order to generate a reduced-dimensional representation of $A$.

This spectral decomposition of our linear operator is of particular importance, because it sheds light on the fact that DMD models the temporal dynamics of our system using a Fourier basis. Each spatial mode is represented by a particular Fourier frequency along with a growth-decay constant that determines the future behavior of our spatial mode. Additionally, the Fourier basis also determines what sorts of time series can be modeled using DMD – time series that are expected to have sinusoidal behavior will be more reliably modeled using DMD, whereas signals that show abrupt spike patterns might be more difficult to model.

P.J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics 656.1. 2010. ↩︎
Tu et al. On Dynamic Mode Decomposition: Theory And Applications ↩︎
Kunert-Graf et al. Extracting Reproducible Time-Resolved Resting State Networks Using Dynamic Mode Decomposition. Front. Comput. Neurosci. 2019. ↩︎

Multivariate Normal Distribution

Sat, 12 May 2018 03:14:14 -0700

In this post, I’ll be covering the basics of Multivariate Normal Distributions, with special emphasis on deriving the conditional and marginal distributions.

Given a random variable under the usual Gauss-Markov assumptions, with $y_{i} \sim N(\mu, \sigma^{2})$ with $e \sim N(0,\sigma^{2})$, and $N$ independent samples $y_{1}…y_{n}$, we can define vector $\mathbf{y} = [y_{1}, y_{2},…y_{n}] \sim N_{n}(\mathbf{\mu},\sigma^{2}I)$ with $\mathbf{e} \sim N_{n}(\mathbf{0},\sigma^{2}I)$. We can see from the covariance structure of the errors that all off-diagonal elements are 0, indicating that our samples are independent with equal variances.

Marginal Distributions

Now assume that $\mathbf{y} = [\mathbf{y_{1}}, \mathbf{y_{2}} ] \sim N(\mathbf{\mu},\Sigma)$, where $\mathbf{\mu} = \begin{bmatrix} \mu_{1} \ \mu_{2} \end{bmatrix}$, and $\Sigma$ is an arbitrary covariance matrix, where we cannot assume independence. If $\Sigma$ is non-singular, we can decompose $\Sigma$ as

$$ \Sigma = \begin{bmatrix} \Sigma_{11} & \Sigma_{21}^{T} \\ \Sigma_{21} & \Sigma_{22} \end{bmatrix}$$

and, using the inversion lemmas from Blockwise Matrix Inversion, define its inverse $\Sigma^{-1} = V$ as

$$ V = \begin{bmatrix} V_{11} & V_{21}^{T} \\ V_{21} & V_{22} \\ \end{bmatrix} \begin{bmatrix} (\Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21})^{-1} & -\Sigma^{-1}\Sigma_{12}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1} \\ -\Sigma_{22}^{-1}\Sigma_{21}(\Sigma_{11}-\Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21})^{-1} & (\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1} \end{bmatrix}$$

From the properties of transformations of Normal random variables, we can define the marginal of $$y_{1}$$ as

$$\begin{align} By \sim N(B\mu,B\Sigma B^{T}) \end{align}$$

where $B = \begin{bmatrix} \mathbf{I} & 0 \end{bmatrix}$ such that

$$ \begin{bmatrix} \mathbf{I} & 0 \end{bmatrix} \begin{bmatrix} \mathbf{\mu_{1}} \ \mathbf{\mu_{2}} \end{bmatrix} = \mathbf{\mu_{1}}$$ $$\begin{bmatrix} \mathbf{I} & 0 \end{bmatrix} \begin{bmatrix} \Sigma_{11} & \Sigma_{12} \\ \Sigma_{21} & \Sigma_{22} \end{bmatrix} \begin{bmatrix} \mathbf{I} \\ 0 \end{bmatrix} = \Sigma_{11}$$

so that $\mathbf{y_{1}} \sim N(\mathbf{\mu_{1}},\Sigma_{11})$.

Conditional Distributions

Showing the conditional distribution is a bit long-winded, so bear with me. We are interested in finding the distribution of $y_{2}\mid y_{1}$, which we can explicitly represent as

$$\begin{align} f_{y_{1}}(y_{2} \mid y_{1}) = \frac{f_{y_{1},y_{2}}(y_{1},y_{2})}{f_{y_{1}}(y_{1})} \end{align}$$

Writing out the joint density for $y$, we have the following

$$\begin{align} f(y) = \frac{1}{(2\pi)^{n/2}\mid \Sigma \mid ^{1/2}}\exp^{(-1/2)(y-\mu)^{T}\Sigma^{-1}(y-\mu)} \end{align}$$

Partitioning this expression up into the individual terms related to $y_{1}$ and $y_{2}$, the exponent becomes

$$ (y-\mu)^{T}V(y-\mu) = \begin{bmatrix} y_{1} - \mu_{1} \\ y_{2} - \mu_{2} \end{bmatrix}^{T} \begin{bmatrix} V_{11} & V_{12} \\ V_{21} & V_{22} \end{bmatrix} \begin{bmatrix} y_{1} - \mu_{1} \\ y_{2} - \mu_{2} \end{bmatrix}$$

Expanding this quadratic form out, we see that we end up with

$$\begin{align} (y_{1} - \mu_{1})^{T} V_{11}^{-1}(y_{1}-\mu_{1}) + 2(y_{1}-\mu_{1})^{T}V_{12}(y_{2}-\mu_{2}) + (y_{2} - \mu_{2})^{T}V_{22}(y_{2}-\mu_{2}) \end{align}$$

Let us, for simplicity, set $z_{1} = (y_{1} - \mu_{1})$ and $z_{2} = (y_{2} - \mu_{2})$. Substituting back in our definitions of $V_{11}$,$V_{12}$,$V_{21}$, and $V_{22}$, and using the Sherman-Morrison-Woodbury definition for $V_{11}$, we have the following

$$\begin{align} &z_{1}^{T}(\Sigma_{11}^{-1} + \Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1}\Sigma_{21}\Sigma_{11})z_{1} \\ &- 2z_{1}^{T}(\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}^{-1})^{-1})z_{2} \\ &+ z_{2}^{T}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1})z_{2} \end{align}$$

which, by distribution of $z_{1}$ across the first term and splitting the second term into its two sums, we have

$$\begin{align} &z_{1}^{T}\Sigma_{11}^{-1}z_{11} + z_{1}^{T}\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22} - \Sigma_{21}V_{11}^{-1}\Sigma_{12})^{-1}\Sigma_{21}\Sigma_{11}^{-1}z_{1} \\ &- z_{1}^{T}(\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1})z_{2} - z_{1}^{T}(\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1})z_{2} \\ &+ z_{2}^{T}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1})z_{2} \end{align}$$

We can pull out forms $z_{1}^{T}\Sigma_{11}^{-1}\Sigma_{12}(\Sigma_{22}-\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1}$ to the left and $(\Sigma_{22}-\Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1}z_{2}$ to the right and, after applying a transpose, have

$$\begin{align} =z_{1}^{T}\Sigma_{11}^{-1}z_{11} + (z_{2} -\Sigma_{21}\Sigma_{11}^{-1}z_{1})^{T}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1}(z_{2} - \Sigma_{21}\Sigma_{11}^{-1}z_{1}) \end{align}$$

Plugging the above back into our exponential term in our original density function, we see that we have a product of two exponential terms

$$\begin{align} &\frac{1}{C_{1}} \exp(\frac{-1}{2}(z_{1}^{T}\Sigma_{11}^{-1}z_{11})) \\ \end{align}$$

and

$$\begin{align} &\frac{1}{C_{2}}\exp(\frac{-1}{2}(z_{2} - z_{1}\Sigma_{11}^{-1}\Sigma_{12})^{T}(\Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12})^{-1}(z_{2} - \Sigma_{21}\Sigma_{11}^{-1}z_{1})) \end{align}$$

where

$$\begin{align} C_{1} &= (2\pi)^{p/2}\mid \Sigma_{11} \mid^{1/2} \\ C_{2} &= (2\pi)^{q/2}\mid \Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12} \mid ^{1/2} \end{align}$$

The first term is the marginal density of $y_{1}$ and the second is the conditional density of $y_{2} \mid y_{1}$ with conditional mean $\mu_{2\mid 1} = \mu_{2} + \Sigma_{11}^{-1}\Sigma_{12}(y_{1} - \mu_{1})$ and conditional variance $\Sigma_{2\mid 1} = \Sigma_{22} - \Sigma_{21}\Sigma_{11}^{-1}\Sigma_{12}$.

While long and drawn out, the formulas show that the conditional distribution of any subset of Normal random variables, given another subset, is also a Normal distribution, with conditional mean and variance defined by functions of the means and covariances of the original random vector.

Rank One Updates

Fri, 11 May 2018 16:53:45 -0700

In this post, I’m going to go over some examples of rank-one updates of matrices. To compute rank-one updates, we rely on the Sherman-Morrison-Woodbury theorem. From the previous post on Blockwise Matrix Inversion, recall that, given a matrix and its inverse

$$R = \begin{bmatrix} A & B \\ C & D \end{bmatrix} \; \; \; \; R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix}$$

we have that

$$\begin{align} W = (A-BD^{-1}C)^{-1} = C^{-1}D(D-CA^{-1}B)^{-1}CA^{-1} \end{align}$$

Expanding this further, the Woodbury formula proves the following identity

$$\begin{align} (A+BD^{-1}C)^{-1}=A^{-1}-A^{-1}B(D−CA^{-1}B)^{-1}CA^{-1} \end{align}$$

Given an initial matrix $A$ and its inverse $A^{-1}$, and a new matrix $R=BD^{-1}C$, we see that we can define the inverse of our new updated matrix $A+R$ in terms of the inverse of our original matrix $A$ and components of $R$. Importantly, we can perform rank-$k$ updates, where $rank(R) = k$.

For example, if we want to update our matrix $A$ with a new vector, $v$, we can rewrite the formula above as follows:

$$\begin{align} (A+vv^{T})^{-1} &=A^{-1}-A^{-1}v(1+v^{T}A^{-1}v)^{-1}v^{T}A^{-1} \\ &=A^{-1}-\frac{A^{-1}vv^{T}A^{-1}}{1+v^{T}A^{-1}v} \\ \end{align}$$

where the updated inverse is defined so long as the quadratic form $v^{T}A^{-1}v \neq -1$.

Rank-One Updates for Linear Models

Recall the Normal equations for linear models:

$$\begin{align} X^{T}X\beta = X^{T}y \end{align}$$

and

$$\begin{align} \beta = (X^{T}X)^{g}X^{T}y \end{align}$$

where $X$ is our design matrix, $y$ is our dependent variable, and $\beta$ is a solution to the Normal equation, due to the fact that the Normal equations are consistent. $(X^{T}X)^{g}$ is the generalized inverse of $X^{T}X$, which is unique (i.e. $(X^{T}X)^{g} = (X^{T}X)^{-1}$) only if $X$ has full column-rank. For our immediate purpose, we assume that $X$ has full column rank.

Assume that we observe a set of observations, $X \in R^{n \times p}$ and response variable, $y$, and compute our coefficient estimates $\hat{\beta}$ via the Normal equations above, using $(X^{T}X)^{-1}$. Now given a new observation, $v \in R^{p}$, how can we update our coefficient estimates? We can append $v$ to $X$ as

$$ X^{\text{*}} = \begin{bmatrix} X \\ v \end{bmatrix} \in R^{(n+1) \times p}$$

and directly compute $(X^{\text{*T}}X^{\text{*}})^{-1}$, or we can use the Sherman-Morrison-Woodbury theorem:

$$ \begin{align} (X^{\text{*T}}X^{\text{*}})^{-1} = (X^{T}X + vv^{T})^{-1} = (X^{T}X)^{-1} - \frac{(X^{T}X)^{-1}vv^{T}(X^{T}X)^{-1}}{1+v^{T}(X^{T}X)^{-1}v} \\ \end{align} $$

from which we can easily compute our new coefficient estimates with $$.

$$\begin{align} \beta^{\text{*}} = (X^{\text{*T}}X^{\text{*}})^{-1}X^{\text{*}T}y \\ \end{align}$$

Importantly, in the case of regression, for example, this means that we can update our linear model via simple matrix calculations, rather than having to refit the model from scratch to incorporate our new data. In the next few posts, I’ll go over an example of an implementation of rank-updating methods that I’ve been using in lab to study brain dynamics.

Blockwise Matrix Inversion

Tue, 08 May 2018 23:24:17 -0700

I’m taking a Statistics course on the theory of linear models, which covers Gauss-Markov models and various extensions of them. Sometimes, when dealing with partitioned matrices, and commonly Multivariate Normal Distributions, we’ll often need to invert matrices in a blockwise manner. This has happened often enough during this course (coincidentally was necessary knowledge for a midterm question), so I figured I should just document some of the inversion lemmas.

Let’s define our partitioned matrix as

$$ R = \begin{bmatrix} A & B \\ C & D \end{bmatrix}$$

We are specifically interested in finding

$$ R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix}$$

such that

$$ R R^{-1} = R^{-1}R = \begin{bmatrix} I & 0 \\ 0 & I \end{bmatrix}$$

Part 1: $R R^{-1}$

For the right inverse ($R R^{-1}$), we can define

$$ \begin{aligned} AW + BY = I \\ AX + BZ = 0 \\ CW + DY = 0 \\ CX + DZ = I \\ \end{aligned} $$

and, assuming $A$ and $D$ are invertible,

$$\begin{aligned} X = -A^{-1}BZ \\ Y = -D^{-1}CW \\ \end{aligned}$$

We can plug these identities back into the first system of equations as

$$\begin{aligned} AW + B(-D^{-1}CW) &= (A - BD^{-1}C)W = I \\ C(-A^{-1}BZ) + DZ &= (D - CA^{-1}B)Z = I \\ \end{aligned}$$

so that

$$\begin{aligned} W = (A-BD^{-1}C)^{-1} \\ Z = (D-CA^{-1}B)^{-1} \\ \end{aligned}$$

and finally

$$ R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix} = \begin{bmatrix} (A-BD^{-1}C)^{-1} & -A^{-1}B(D-CA^{-1}B)^{-1} \\ -D^{-1}C(A-BD^{-1}C)^{-1} & (D-CA^{-1}B)^{-1} \\ \end{bmatrix}$$

It is important to note that the above result only holds if $A$, $D$, $(D-CA^{-1}B)$, and $(A-BD^{-1}C)$ are invertible.

Part 2: $R^{-1} R$

Following the same logic as above, we have the following systems of equations for the left inverse ($R^{-1}R$)

$$\begin{aligned} WA + XC = I \\ WB + XD = 0 \\ YA + ZC = 0 \\ YB + ZD = I \\ \end{aligned}$$

so that

$$\begin{aligned} X = WBD^{-1} = A^{-1}BZ \\ Y = ZCA^{-1} = D^{-1}CW \\ \end{aligned}$$

which indicates that

$$\begin{aligned} W = (A-BD^{-1}C)^{-1} = C^{-1}D(D-CA^{-1}B)^{-1}CA^{-1} \\ X = (A-BD^{-1}C)^{-1}BD^{-1} = A^{-1}B(D-CA^{-1}B)^{-1} \\ \end{aligned}$$

Importantly, blockwise matrix inversion allows us to define the inverse of a larger matrix, with respect to its subcomponents. Likewise, from here, we can go on to derive the Sherman-Morrison formula and Woodbury theorem, which allow us to do all kinds of cool stuff, like rank-one matrix updates. In the next few posts, I’ll go over a few examples of where blockwise matrix inversions are useful, and common scenarios where rank-one updates of matrices are applicable.