Rank One Updates

Fri, 11 May 2018 16:53:45 -0700

In this post, I’m going to go over some examples of rank-one updates of matrices. To compute rank-one updates, we rely on the Sherman-Morrison-Woodbury theorem. From the previous post on Blockwise Matrix Inversion, recall that, given a matrix and its inverse

$$R = \begin{bmatrix} A & B \\ C & D \end{bmatrix} \; \; \; \; R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix}$$

we have that

$$\begin{align} W = (A-BD^{-1}C)^{-1} = C^{-1}D(D-CA^{-1}B)^{-1}CA^{-1} \end{align}$$

Expanding this further, the Woodbury formula proves the following identity

$$\begin{align} (A+BD^{-1}C)^{-1}=A^{-1}-A^{-1}B(D−CA^{-1}B)^{-1}CA^{-1} \end{align}$$

Given an initial matrix $A$ and its inverse $A^{-1}$, and a new matrix $R=BD^{-1}C$, we see that we can define the inverse of our new updated matrix $A+R$ in terms of the inverse of our original matrix $A$ and components of $R$. Importantly, we can perform rank-$k$ updates, where $rank(R) = k$.

For example, if we want to update our matrix $A$ with a new vector, $v$, we can rewrite the formula above as follows:

$$\begin{align} (A+vv^{T})^{-1} &=A^{-1}-A^{-1}v(1+v^{T}A^{-1}v)^{-1}v^{T}A^{-1} \\ &=A^{-1}-\frac{A^{-1}vv^{T}A^{-1}}{1+v^{T}A^{-1}v} \\ \end{align}$$

where the updated inverse is defined so long as the quadratic form $v^{T}A^{-1}v \neq -1$.

Rank-One Updates for Linear Models

Recall the Normal equations for linear models:

$$\begin{align} X^{T}X\beta = X^{T}y \end{align}$$

and

$$\begin{align} \beta = (X^{T}X)^{g}X^{T}y \end{align}$$

where $X$ is our design matrix, $y$ is our dependent variable, and $\beta$ is a solution to the Normal equation, due to the fact that the Normal equations are consistent. $(X^{T}X)^{g}$ is the generalized inverse of $X^{T}X$, which is unique (i.e. $(X^{T}X)^{g} = (X^{T}X)^{-1}$) only if $X$ has full column-rank. For our immediate purpose, we assume that $X$ has full column rank.

Assume that we observe a set of observations, $X \in R^{n \times p}$ and response variable, $y$, and compute our coefficient estimates $\hat{\beta}$ via the Normal equations above, using $(X^{T}X)^{-1}$. Now given a new observation, $v \in R^{p}$, how can we update our coefficient estimates? We can append $v$ to $X$ as

$$ X^{\text{*}} = \begin{bmatrix} X \\ v \end{bmatrix} \in R^{(n+1) \times p}$$

and directly compute $(X^{\text{*T}}X^{\text{*}})^{-1}$, or we can use the Sherman-Morrison-Woodbury theorem:

$$ \begin{align} (X^{\text{*T}}X^{\text{*}})^{-1} = (X^{T}X + vv^{T})^{-1} = (X^{T}X)^{-1} - \frac{(X^{T}X)^{-1}vv^{T}(X^{T}X)^{-1}}{1+v^{T}(X^{T}X)^{-1}v} \\ \end{align} $$

from which we can easily compute our new coefficient estimates with $$.

$$\begin{align} \beta^{\text{*}} = (X^{\text{*T}}X^{\text{*}})^{-1}X^{\text{*}T}y \\ \end{align}$$

Importantly, in the case of regression, for example, this means that we can update our linear model via simple matrix calculations, rather than having to refit the model from scratch to incorporate our new data. In the next few posts, I’ll go over an example of an implementation of rank-updating methods that I’ve been using in lab to study brain dynamics.

Blockwise Matrix Inversion

Tue, 08 May 2018 23:24:17 -0700

I’m taking a Statistics course on the theory of linear models, which covers Gauss-Markov models and various extensions of them. Sometimes, when dealing with partitioned matrices, and commonly Multivariate Normal Distributions, we’ll often need to invert matrices in a blockwise manner. This has happened often enough during this course (coincidentally was necessary knowledge for a midterm question), so I figured I should just document some of the inversion lemmas.

Let’s define our partitioned matrix as

$$ R = \begin{bmatrix} A & B \\ C & D \end{bmatrix}$$

We are specifically interested in finding

$$ R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix}$$

such that

$$ R R^{-1} = R^{-1}R = \begin{bmatrix} I & 0 \\ 0 & I \end{bmatrix}$$

Part 1: $R R^{-1}$

For the right inverse ($R R^{-1}$), we can define

$$ \begin{aligned} AW + BY = I \\ AX + BZ = 0 \\ CW + DY = 0 \\ CX + DZ = I \\ \end{aligned} $$

and, assuming $A$ and $D$ are invertible,

$$\begin{aligned} X = -A^{-1}BZ \\ Y = -D^{-1}CW \\ \end{aligned}$$

We can plug these identities back into the first system of equations as

$$\begin{aligned} AW + B(-D^{-1}CW) &= (A - BD^{-1}C)W = I \\ C(-A^{-1}BZ) + DZ &= (D - CA^{-1}B)Z = I \\ \end{aligned}$$

so that

$$\begin{aligned} W = (A-BD^{-1}C)^{-1} \\ Z = (D-CA^{-1}B)^{-1} \\ \end{aligned}$$

and finally

$$ R^{-1} = \begin{bmatrix} W & X \\ Y & Z \end{bmatrix} = \begin{bmatrix} (A-BD^{-1}C)^{-1} & -A^{-1}B(D-CA^{-1}B)^{-1} \\ -D^{-1}C(A-BD^{-1}C)^{-1} & (D-CA^{-1}B)^{-1} \\ \end{bmatrix}$$

It is important to note that the above result only holds if $A$, $D$, $(D-CA^{-1}B)$, and $(A-BD^{-1}C)$ are invertible.

Part 2: $R^{-1} R$

Following the same logic as above, we have the following systems of equations for the left inverse ($R^{-1}R$)

$$\begin{aligned} WA + XC = I \\ WB + XD = 0 \\ YA + ZC = 0 \\ YB + ZD = I \\ \end{aligned}$$

so that

$$\begin{aligned} X = WBD^{-1} = A^{-1}BZ \\ Y = ZCA^{-1} = D^{-1}CW \\ \end{aligned}$$

which indicates that

$$\begin{aligned} W = (A-BD^{-1}C)^{-1} = C^{-1}D(D-CA^{-1}B)^{-1}CA^{-1} \\ X = (A-BD^{-1}C)^{-1}BD^{-1} = A^{-1}B(D-CA^{-1}B)^{-1} \\ \end{aligned}$$

Importantly, blockwise matrix inversion allows us to define the inverse of a larger matrix, with respect to its subcomponents. Likewise, from here, we can go on to derive the Sherman-Morrison formula and Woodbury theorem, which allow us to do all kinds of cool stuff, like rank-one matrix updates. In the next few posts, I’ll go over a few examples of where blockwise matrix inversions are useful, and common scenarios where rank-one updates of matrices are applicable.

Matrix Inversion |

Rank One Updates

Blockwise Matrix Inversion