Why I’m learning Kubernetes for ML engineering, and getting two services talking on a local cluster.
We are a pretty small team, yet we still have a pretty large and expansive suite of services, dashboards, and backend data systems. When the number of services was small, auditting and logging was quite easy, but over the last few years, it’s become quite a chore. Imagine the following scenario:
I’m building tools that help scientists run their day-to-day work. I work in a heavily regulated environment, subject to both U.S. and European digital policy requirements. The scenario we face regularly is: two services talk to each other. Then three. A dashboard calls a backend API, that API calls another one, a nightly job calls all of them. Somewhow, we have to answer how one service proves to another that it is allowed to make the call.
Here is the second of two posts on running Dagster in ECS. The first dealt with the platform side: the Daemon, the two Webservers, the ALB and Cognito wiring, the three security groups, the Cloud Map namespace, and the IAM that lets the Daemon launch anything at all.
Here at Just-Evotec Biologics, the Data Platform team uses Dagster as our orchestration platform of choice. Many of our services are deployed to AWS ECS as Fargate or EC2 tasks. Getting Dagster onto ECS took me an embarrassing number of hours, and most of those issues stemmed from a lack of documentation around getting Dagster pipelines up and running in the cloud while NOT using Dagster+ (their premium service offering). The official docs cover the happy, perfect world path, but beyond that, you’re left stitching together forum threads, GitHub issues, and a lot of trial and error. I figured I’d put together some notes on what I built, and how I built it.
When I joined Just-Evotec Biologics (first as a data scientist, now as a data platform engineer), I inherited a data ecosystem that probably looked pretty familiar to what lots of others have dealth with: a dozen scientific instruments each outputting data in proprietary formats, a LIMS system with its own schema, upstream and downstream experimental systems, Excel workbooks scientists had been maintaining for years but hidden away from any production storage system, and a swathe of applications that had been developed previously but had very little in the way of data governance. We didn’t have a central schema definitions, very little in the way of contracts between producers (e.g. instrumentation) and consumers (e.g. scientists, applications, clients, more instruments), and no way to ask or answer “what’s being pulled down here?”.
This is the second of two posts on designing Gitlab CI/CD pipelines. The
first covered the anatomy of a .gitlab-ci.yml file, conditional jobs, and setting up the .pypirc and .netrc files that let a pipeline build a Python package and push it to a registry.
I recently developed a template workflow to help our team adopt a CI/CD-based development strategy. Many of our web applications and tools were based on simple repository structures. With growing datasets and ever-increasing use by outside teams, we found ourselves needing to add new features more frequently to many of these tools and believed that continuous integration and deployment could help us not just develop more quickly, but also more intelligently. Since we use Gitlab to store our code, we decided to use the Gitlab CI/CD tools.
I’m interested in looking at some spatial mappings between pairs of cortical regions, and believe that these mappings are mediated, to some degree, by the temporal coupling between cortical areas. I don’t necessarily know the functional form of these mappings, but neurobiologically predict that these mappings are not random and have some inherent structure. I want to examine the relationship between spatial location and strength of temporal coupling. I’m going to use mutual information to measure this association.
As I mentioned in my previous post on constrained graph attention networks, graph neural networks suffer from overfitting and oversmoothing as network depth increases. These issues can ultimately be linked to the local topologies of the graph.