Linear Algebra |

Exploring Neurological Dynamical Systems: Part 2

Thu, 24 May 2018 11:30:43 -0700

In my previous post on dynamic mode decomposition, I discussed the foundations of DMD as a means for linearizing a dynamical system¹²³. In this post, I want to look at a way in which we can use rank-updates to incorporate new information into the spectral decomposition of our linear operator, $A$, in the event that we are generating online measurements from our dynamical system⁴ – see the citation below if you want a more-detailed overview of this topic along with open source code for testing this method.

Recall that we are given an initial data matrix

$$\begin{align} X = \begin{bmatrix} x_{n_{1},m_{1}} & x_{n_{1},m_{2}} & x_{n_{1},m_{3}} & … \\ x_{n_{2},m_{1}} & x_{n_{1},m_{2}} & x_{n_{2},m_{3}} & … \\ x_{n_{3},m_{1}} & x_{n_{1},m_{2}} & x_{n_{3},m_{3}} & … \\ … & … & … & … \\ \end{bmatrix} \in R^{n \times m} \end{align}$$

which we can split into two matrices, shifted one unit in time apart:

$$\begin{align} X^{\ast} &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_1 & \vec{x}_2 & \dots & \vec{x}_{m-1} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \\ Y &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_2 & \vec{x}_3 & \dots & \vec{x}_{m} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \end{align}$$

and we are interested in solving for the linear operator $A$, such that

$$\begin{align} Y = AX^{\ast} \end{align}$$

For simplicity, since we are no longer using the full matrix, I’ll just refer to $X^{\ast}$ as $X$. In the previous post, we made the constraint that $n > m$, and that rank($X$) $\leq m < n$. Here, however, we’ll reverse this assumption, such that $m > n$, and that rank($X$) $\leq m < n$, such that $XX^{T}$ is invertible, so by multiplying both sides by $X^{T}$ we have

$$\begin{align} AXX^{T} &= YX^{T} \\ A &= YX^{T}(XX^{T})^{-1} \\ A &= QP_{x} \end{align}$$

where $Q = YX^{T}$ and $P_{x} = (XX^{T})^{-1}$. Now, let’s say you observe some new data $x_{m+1}, y_{m+1}$, and you want to incorporate this new data into your $A$ matrix. As in the previous post on rank-one updates, we saw that directly computing the inverse could potentially be costly, so we want to refrain from doing that if possible. Instead, we’ll use the Sherman-Morrison-Woodbury theorem again to incorporate our new $x_{m+1}$ sample into our inverse matrix, just as before:

$$\begin{align} (X_{m+1}X^{T}{m+1})^{-1} = P{x} + \frac{P_{x}x_{m+1}x_{m+1}^{T}P_{x}}{1 + x_{m+1}^{T}P_{x}x_{m+1}} \end{align}$$

Likewise, since we’re appending new data to our $Y$ and $X$ matrices, we also have

$$\begin{align} Y_{m+1} = \begin{bmatrix} Y & y_{m+1} \end{bmatrix} \\ X_{m+1} = \begin{bmatrix} \\ X & x_{m+1} \end{bmatrix} \\ \end{align}$$

such that

$$\begin{align} Y_{m+1} X_{m+1}^{T} &= YX^{T} + y_{m+1}x_{m+1}^{T} \\ &= Q + y_{m+1}x_{m+1}^{T} \end{align}$$

which is simply the sum of our original matrix $Q$, plus a rank-one matrix. The authors go on to describe some pretty cool “local” DMD schemes, by incorporating weights, as well as binary thresholds, that are time-dependent into the computation of the linear operator, $A$.

P.J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics 656.1. 2010. ↩︎
Tu et al. On Dynamic Mode Decomposition: Theory And Applications ↩︎
Kunert-Graf et al. Extracting Reproducible Time-Resolved Resting State Networks Using Dynamic Mode Decomposition. Front. Comput. Neurosci. 2019. ↩︎
Zhang et al. Online dynamic mode decomposition for time-varying systems. 2017. ↩︎

Exploring Neurological Dynamical Systems: Part 1

Tue, 22 May 2018 14:02:52 -0700

In the next two posts, I want to talk briefly about an algorithm called Dynamic Mode Decomposition (DMD). DMD is a spatiotemporal modal decomposition technique that can be used to identify spatial patterns in a signal (modes), along with the time course of these spatial patterns (dynamics). As such, the algorithm assumes that the input data has both a spatial and a temporal component. We are interested in modeling how the system evolves over time.

If you’d like to find more information about DMD, Peter Schmid¹ and Jonathan Tu² have written excellent expositions on the topic. Likewise, if you’d like to follow along with the code for the following analysis, see my repo. For a more in-depth analysis that applies DMD to brain activity in the resting brain, see a recent publication by my colleagues and me³, along with the code used for our analysis.

The DMD Algorithm

Let’s assume that you’ve taken $n$ measurements from specific points in space for $m$ time points, where for now we assume that $m\lt n$. For now, we’ll assume that the sampling frequency, $\omega$, is stable across the entire experiment. We define our entire data matrix as

We are interested in solving for the matrix, $A \in R^{n \times n}$, such that

$$\begin{align} x_{t+1} = A x_{t} \; \; \forall \; \; t = 1,2,…m-1 \end{align}$$

Given our full data matrix $X$, we can define two matrices $X^{*}$ and $Y$ such that

$$\begin{align} X^{*} &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_{1} & \vec{x}_{2} & \dots & \vec{x}_{m-1} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \\ Y &= \begin{bmatrix} \vert & \vert & \dots & \vert \\ \vec{x}_2 & \vec{x}_3 & \dots & \vec{x}_{m} \\ \vert & \vert & \dots & \vert \\ \end{bmatrix} \in R^{n \times (m-1)} \end{align}$$

so that we can write

$$\begin{align} Y = AX^{\ast} \end{align}$$

If $n$ is small, this is relatively easy to compute – however, if $n$ is large, as is the case when modeling temporal dynamics in resting-state MRI, it would be computationally inefficient to compute A directly. To alleviate this, we can make use of the Singular Value Decomposition (SVD) of our predictor matrix $X^{\ast}$. We define the SVD of $X^{\ast}$ as

$$\begin{align} X^{\ast} = U \Sigma V^{T} \ \end{align}$$

as well as the Moore-Penrose pseudo-inverse of $X^{\ast} = X^{\dagger}$ as

$$\begin{align} X^{\dagger} = V \Sigma^{-1} U^{T} \\ \end{align}$$

such that we can write

$$\begin{align} YX^{\dagger} = YV \Sigma^{-1} U^{T} = A X^{\ast}X^{\dagger} = A \\ \end{align}$$

Additionally, if we assume that $rank(X^{\ast}) = r \leq m$, then we can use the truncated SVD such that

$$\begin{align} U & \in R^{n \times r} \\ V^{T} & \in R^{r \times m} \\ \end{align}$$

and

$$\begin{align} \Sigma = \begin{bmatrix} \sigma_{1} & 0 & 0 & … \\ 0 & \sigma_{2} & 0 & … \\ 0 & 0 & \ddots & … \\ \vdots & \vdots & \vdots & \sigma_{r} \\ \end{bmatrix} \in R^{r \times r} \end{align}$$

As it stands now, we still compute an $A \in R^{n \times n}$ matrix. However, because we have a potentially low-rank system, we can apply a Similarity Transformation to $A$ in order to reduce its dimensionality, without changing its spectrum. Using our spatial singular vectors $U$, we define

$$\begin{align} \tilde{A} &= U^{T} A U \\ &= U^{T} (YV \Sigma^{-1} U^{T}) U \\ &= U^{T} Y V \Sigma^{-1} \\ \end{align}$$

where $\tilde{A} \in R^{r \times r}$. If we consider the above SVD, we see that $U$ is the matrix of left singular vectors, an orthogonal basis that spans $C(X^{\ast})$, which is an r-dimensional subspace of $R^{n}$. Thus, the similarity transform represents a mapping $f(A) = U^{T} A U : R^{n} \rightarrow R^{r}$. We now have a reduced-dimensional representation of our linear operator, from which we can compute the spatial modes and dynamic behavior of each mode. First, however, because of the notion of variance captured by the singular values of our original predictor matrix, we weight $\tilde{A}$ by the singular values as

$$\begin{align} \hat{A} = \Sigma^{\frac{1}{2}} \tilde{A} \Sigma^{\frac{1}{2}} \\ \end{align}$$

such that our computed spatial modes have been weighted by the amount they contribute to our measured signal. We can now compute the eigendecomposition of $\hat{A}$ as

$$\begin{align} \hat{A} W = W \Lambda \\ \end{align}$$

where the eigenvectors $W$ are the reduced-dimension representations of our spatial modes, and the eigenvalues $\Lambda$ capture the dynamic behavior of our spatial modes. Because our original data matrix $X^{\ast}$ had spatial dimension $n$ and our eigenvectors have dimension $r$, we need to up-project our eigenvectors $W$ to compute the final spatial modes, via

$$\begin{align} \Phi = Y V \Sigma^{\frac{-1}{2}}W \end{align}$$

From the SVD of our prediction matrix $X^\ast=U \Sigma V^{T}$, the matrix $V \in R^{m \times r}$ is the matrix of right singular vectors, an orthogonal basis spanning the space of $X^{\ast T}$ (i.e. $r$ basis vectors spanning the space of the measured time courses). Thus, we see that $H = (V \Sigma^{\frac{-1}{2}})W$ represents a linear combination of the temporal basis vectors (a mapping from $R^{r} \rightarrow R^{m}$) for each eigenvector $w_{i}$ of $W$, weighted by the corresponding singular value $\sigma_{i}^{\frac{-1}{2}}$ (that acts to normalize the spatial mode amplitudes). Finally, we see that $\Phi = X^{\ast}H$ computes how much of each temporal basis vector is present in the measured time course at each point in space.

Because we are modeling a dynamical system, we can compute the continuous time dynamics of our system using our spatial modes and eigenvalues as

$$\begin{align} \vec{x}(t) \approx \sum_{i=1}^{r} b_{i}\exp^{((\gamma_{i} + 2i\pi f_{i})\cdot t)} \vec{\phi}_{i} \end{align}$$

where $\gamma_{i}$ is a growth-decay constant and $f_{i}$ is the frequency of oscillation of the spatial mode $\phi_{i}$. We can compute these two constants as

$$\begin{align} \gamma_{i} &= \frac{\text{real}(\text{ln}(\lambda_{i}))}{\Delta t} \\ f_{i} &= \frac{\text{imag}(\text{ln}(\lambda_{i}))}{2\pi \Delta t} \end{align}$$

So, we can see that DMD linearizes our measured time series, by fitting what can be analogized to a “global” regression. That is, instead of computing how a single time point predicts the next time point, which could readily be solved using the simple Normal equations, DMD computes how a matrix of time points predicts another matrix of time points that is shifted one unit of time into the future. To this extent, DMD minimizes the Frobenius norm of

$$\begin{align} \min \limits_{A} \lVert Y - AX^{\ast} \rVert^{2}_{F} \\ \end{align}$$

However, rather than explicitly computing the matrix $A$, DMD computes the eigenvectors and eigenvalues of $A$, by utilizing the Singular Value Decomposition, along with a Similarity Transformation, in order to generate a reduced-dimensional representation of $A$.

This spectral decomposition of our linear operator is of particular importance, because it sheds light on the fact that DMD models the temporal dynamics of our system using a Fourier basis. Each spatial mode is represented by a particular Fourier frequency along with a growth-decay constant that determines the future behavior of our spatial mode. Additionally, the Fourier basis also determines what sorts of time series can be modeled using DMD – time series that are expected to have sinusoidal behavior will be more reliably modeled using DMD, whereas signals that show abrupt spike patterns might be more difficult to model.

P.J. Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics 656.1. 2010. ↩︎
Tu et al. On Dynamic Mode Decomposition: Theory And Applications ↩︎
Kunert-Graf et al. Extracting Reproducible Time-Resolved Resting State Networks Using Dynamic Mode Decomposition. Front. Comput. Neurosci. 2019. ↩︎

Rank One Updates

Fri, 11 May 2018 16:53:45 -0700

In this post, I’m going to go over some examples of rank-one updates of matrices. To compute rank-one updates, we rely on the Sherman-Morrison-Woodbury theorem. From the previous post on Blockwise Matrix Inversion, recall that, given a matrix and its inverse

$$R = \begin{bmatrix} A & B \\ C & D \end{bmatrix} \; \; \; \; R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix}$$

we have that

$$\begin{align} W = (A-BD^{-1}C)^{-1} = C^{-1}D(D-CA^{-1}B)^{-1}CA^{-1} \end{align}$$

Expanding this further, the Woodbury formula proves the following identity

$$\begin{align} (A+BD^{-1}C)^{-1}=A^{-1}-A^{-1}B(D−CA^{-1}B)^{-1}CA^{-1} \end{align}$$

Given an initial matrix $A$ and its inverse $A^{-1}$, and a new matrix $R=BD^{-1}C$, we see that we can define the inverse of our new updated matrix $A+R$ in terms of the inverse of our original matrix $A$ and components of $R$. Importantly, we can perform rank-$k$ updates, where $rank(R) = k$.

For example, if we want to update our matrix $A$ with a new vector, $v$, we can rewrite the formula above as follows:

$$\begin{align} (A+vv^{T})^{-1} &=A^{-1}-A^{-1}v(1+v^{T}A^{-1}v)^{-1}v^{T}A^{-1} \\ &=A^{-1}-\frac{A^{-1}vv^{T}A^{-1}}{1+v^{T}A^{-1}v} \\ \end{align}$$

where the updated inverse is defined so long as the quadratic form $v^{T}A^{-1}v \neq -1$.

Rank-One Updates for Linear Models

Recall the Normal equations for linear models:

$$\begin{align} X^{T}X\beta = X^{T}y \end{align}$$

and

$$\begin{align} \beta = (X^{T}X)^{g}X^{T}y \end{align}$$

where $X$ is our design matrix, $y$ is our dependent variable, and $\beta$ is a solution to the Normal equation, due to the fact that the Normal equations are consistent. $(X^{T}X)^{g}$ is the generalized inverse of $X^{T}X$, which is unique (i.e. $(X^{T}X)^{g} = (X^{T}X)^{-1}$) only if $X$ has full column-rank. For our immediate purpose, we assume that $X$ has full column rank.

Assume that we observe a set of observations, $X \in R^{n \times p}$ and response variable, $y$, and compute our coefficient estimates $\hat{\beta}$ via the Normal equations above, using $(X^{T}X)^{-1}$. Now given a new observation, $v \in R^{p}$, how can we update our coefficient estimates? We can append $v$ to $X$ as

$$ X^{\text{*}} = \begin{bmatrix} X \\ v \end{bmatrix} \in R^{(n+1) \times p}$$

and directly compute $(X^{\text{*T}}X^{\text{*}})^{-1}$, or we can use the Sherman-Morrison-Woodbury theorem:

$$ \begin{align} (X^{\text{*T}}X^{\text{*}})^{-1} = (X^{T}X + vv^{T})^{-1} = (X^{T}X)^{-1} - \frac{(X^{T}X)^{-1}vv^{T}(X^{T}X)^{-1}}{1+v^{T}(X^{T}X)^{-1}v} \\ \end{align} $$

from which we can easily compute our new coefficient estimates with $$.

$$\begin{align} \beta^{\text{*}} = (X^{\text{*T}}X^{\text{*}})^{-1}X^{\text{*}T}y \\ \end{align}$$

Importantly, in the case of regression, for example, this means that we can update our linear model via simple matrix calculations, rather than having to refit the model from scratch to incorporate our new data. In the next few posts, I’ll go over an example of an implementation of rank-updating methods that I’ve been using in lab to study brain dynamics.

Blockwise Matrix Inversion

Tue, 08 May 2018 23:24:17 -0700

I’m taking a Statistics course on the theory of linear models, which covers Gauss-Markov models and various extensions of them. Sometimes, when dealing with partitioned matrices, and commonly Multivariate Normal Distributions, we’ll often need to invert matrices in a blockwise manner. This has happened often enough during this course (coincidentally was necessary knowledge for a midterm question), so I figured I should just document some of the inversion lemmas.

Let’s define our partitioned matrix as

$$ R = \begin{bmatrix} A & B \\ C & D \end{bmatrix}$$

We are specifically interested in finding

$$ R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix}$$

such that

$$ R R^{-1} = R^{-1}R = \begin{bmatrix} I & 0 \\ 0 & I \end{bmatrix}$$

Part 1: $R R^{-1}$

For the right inverse ($R R^{-1}$), we can define

$$ \begin{aligned} AW + BY = I \\ AX + BZ = 0 \\ CW + DY = 0 \\ CX + DZ = I \\ \end{aligned} $$

and, assuming $A$ and $D$ are invertible,

$$\begin{aligned} X = -A^{-1}BZ \\ Y = -D^{-1}CW \\ \end{aligned}$$

We can plug these identities back into the first system of equations as

$$\begin{aligned} AW + B(-D^{-1}CW) &= (A - BD^{-1}C)W = I \\ C(-A^{-1}BZ) + DZ &= (D - CA^{-1}B)Z = I \\ \end{aligned}$$

so that

$$\begin{aligned} W = (A-BD^{-1}C)^{-1} \\ Z = (D-CA^{-1}B)^{-1} \\ \end{aligned}$$

and finally

$$ R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix} = \begin{bmatrix} (A-BD^{-1}C)^{-1} & -A^{-1}B(D-CA^{-1}B)^{-1} \\ -D^{-1}C(A-BD^{-1}C)^{-1} & (D-CA^{-1}B)^{-1} \\ \end{bmatrix}$$

It is important to note that the above result only holds if $A$, $D$, $(D-CA^{-1}B)$, and $(A-BD^{-1}C)$ are invertible.

Part 2: $R^{-1} R$

Following the same logic as above, we have the following systems of equations for the left inverse ($R^{-1}R$)

$$\begin{aligned} WA + XC = I \\ WB + XD = 0 \\ YA + ZC = 0 \\ YB + ZD = I \\ \end{aligned}$$

so that

$$\begin{aligned} X = WBD^{-1} = A^{-1}BZ \\ Y = ZCA^{-1} = D^{-1}CW \\ \end{aligned}$$

which indicates that

$$\begin{aligned} W = (A-BD^{-1}C)^{-1} = C^{-1}D(D-CA^{-1}B)^{-1}CA^{-1} \\ X = (A-BD^{-1}C)^{-1}BD^{-1} = A^{-1}B(D-CA^{-1}B)^{-1} \\ \end{aligned}$$

Importantly, blockwise matrix inversion allows us to define the inverse of a larger matrix, with respect to its subcomponents. Likewise, from here, we can go on to derive the Sherman-Morrison formula and Woodbury theorem, which allow us to do all kinds of cool stuff, like rank-one matrix updates. In the next few posts, I’ll go over a few examples of where blockwise matrix inversions are useful, and common scenarios where rank-one updates of matrices are applicable.