Software Engineering |

Correlation IDs and Request Lineage Across Services

Tue, 27 Jan 2026 09:00:00 -0800

We are a pretty small team, yet we still have a pretty large and expansive suite of services, dashboards, and backend data systems. When the number of services was small, auditting and logging was quite easy, but over the last few years, it’s become quite a chore. Imagine the following scenario:

A scientist tells you the page on the cell culture analysis dashboard was slow this morning.

If you can’t answer at least one of the following, you might find this post helpful:

Which page?
What do they mean “slow”?
Which user?
When?
What endpoint and what parameters?

The dashboard they’re referring to called some backend API, that API called two more, and one of those queried the LIMS system directly. That’s multiple services and log streams, and I’m willing to bet that not one line in any of them tells you which entries belong to that user’s click.

We face this issue in a variety of places in our data platform. I don’t mean logging in the general sense of import logging, but the narrower scope of how a request identifies itself as it moves between all of our services, so that afterwards we can put the pieces back in order and say which step was slow, or which one returned the 403 error.

I built something that I’ve been slowly deploying and incorporating into most of our services and dashboards. Three pieces do the work: a request-scoped context, middleware that fills it in, and a handful of headers forwarded on every outbound call. It pairs with the previous post on service-to-service auth, since knowing who called and when is half the battle.

What I wanted every line to carry

My goal for this tooling was that any single “typical” log line, anywhere in the system, can answer which request, which user, which service, which endpoint, how long, and what happened. This ends up being a fixed set of fields stamped onto every record key:

Field	Example	Why
`request_id`	`req-474c2493-...`	Ties every line from one request together, across services
`session_id`	`ses-a55813da-...`	Ties multiple requests from one browser session together
`user_id`	`user@example.com`	Who was actually doing this
`service`	`lims-adapter`	Which service emitted the line
`requesting_service`	`dashboard`	Which service called this one
`method` / `path`	`GET` `/samples/S-123/molecule`	Which endpoint
`client`	`10.0.5.233`	Source IP
`status_code`	`403`	What happened
`duration_s`	`1.284`	How long it took
`event`	`request_end`	Machine-readable label for the kind of line

From my perspective, request_id and session_id are the most helpful. A request ID covers one call to a service and everything it fans out into. A session ID covers everything one person did in one sitting. When someone says “it was slow this morning”, you find their session, then look at which requests inside it were slow.

How the Context Travels

There are two mechanisms for how context travels within and between requests, one inside a process and one between processes.

graph LR
Browser(Browser)
Dash[Dashboard]
ApiA[Service A]
ApiB[Service B]
Lims[LIMS]
Logs[Log store]
Browser -->|"click"| Dash
Dash -->|"X-Request-ID, X-Session-ID, X-User-ID, X-Requesting-Service: dashboard"| ApiA
ApiA -->|"same IDs, X-Requesting-Service: service-a"| ApiB
ApiA -->|"same IDs"| Lims
Dash -.->|"req-abc"| Logs
ApiA -.->|"req-abc"| Logs
ApiB -.->|"req-abc"| Logs
classDef store fill:#f3e8fd,stroke:#a142f4
class Logs store

Inside a process, the fields live in a ContextVar, so they follow the request through await chains and across threads without being passed as arguments. Between processes, they’re plain HTTP headers that outbound calls forward. X-Requesting-Service is the one that changes at each hop, while the other three carry through unchanged. That is what lets one request_id span the entire tree.

Why use a filter, not a LoggerAdapter

One way to attach context to log records is by using Python’s logging.LoggerAdapter, but I think it’s the wrong tool for what I’m trying to do. An adapter applies only to log calls made through it. Your own code gets the context, but the third-party library that raises the interesting exception does not, because it holds its own plain logging.getLogger(__name__).

A logging.Filter attached to the root handler sees every record in the process, irrespective of where it came from:

_log_context: ContextVar[dict] = ContextVar("log_context", default={})


class ContextFilter(logging.Filter):
 """Inject current context fields onto every LogRecord."""

 def filter(self, record: logging.LogRecord) -> bool:
 for key, value in {**_global_fields, **_log_context.get()}.items():
 if not hasattr(record, key):
 setattr(record, key, value)
 return True

The hasattr check means a field passed explicitly at the call site wins over the passive default context, so a caller can override user_id for one line without unbinding it. And _global_fields is a plain dict rather than a ContextVar, because ContextVar values aren’t inherited by new threads: static configuration like the service name has to live outside the context or it vanishes in a thread pool.

Binding is a context manager that nests:

@contextmanager
def log_context(**fields):
 current = _log_context.get()
 token = _log_context.set({**current, **fields})
 try:
 yield
 finally:
 _log_context.reset(token)

with log_context(request_id=rid, session_id=sid):
 log.info("request received") # two fields
 with log_context(step="validate"):
 log.info("validating") # three
 log.info("continuing") # back to two

The token mechanism restores the previous context exactly, including when the block exits by exception, which a naive save-and-restore gets wrong.

There’s also a version without a scope boundary, for values that resolve partway through a request. Identity is the usual case: you don’t know the user until auth has run, but you want every line after that point to carry it.

async def get_current_user(token: str) -> User:
 user = decode_token(token)
 update_log_context(user_id=user.email) # every later line carries it
 return user

Middleware

The middleware operates once per request: accept or generate the IDs, resolve the user, bind everything, time the request, and echo the IDs back.

request_id = request.headers.get("x-request-id") or f"req-{uuid.uuid4()}"
session_id = request.headers.get("x-session-id") or f"ses-{uuid.uuid4()}"

Accept-or-generate is what enables cross-service correlation. The first service to see a request mints the ID, and every service after that inherits the minted ID from the header. Echoing the request and session IDs back in the response headers means a client, a browser console, or a support ticket can name the exact request, which turns “it was slow this morning” into a string I can now search for in CloudWatch. Identity resolves from whichever source is available, in priority order:

x-amzn-oidc-data, the ALB-injected Cognito JWT, decoded for the email claim. This is a direct browser hit.
X-User-ID, forwarded by an upstream service that already decoded its own JWT.
A dev stub, when APP_ENV is dev, local, or unset.

The second tier carries a user’s identity into services that the user never talks to directly. Service B has no OIDC header, because its caller was service A, not a browser. It can still log which person’s click ultimately caused the call.

Grading by status

request_end is logged at ERROR for 5xx, WARNING for 4xx, and INFO otherwise:

log.log(
 logging.ERROR if status >= 500
 else logging.WARNING if status >= 400
 else logging.INFO,
 "request completed",
 extra={"event": "request_end", "status_code": status, "duration_s": duration_s},
)

403s that explain nothing

A route raises HTTPException(403, detail="missing scope: service-b/write"). The client gets a useful error. The log gets a bare 403 and nothing else. The reason is ordering: FastAPI’s ExceptionMiddleware sits inside the middleware, so by the time the middleware sees anything, the exception has already been converted into a response. The detail explaining the rejection went to the client and nowhere else. At that point, debugging a 403 means reproducing it, which is annoying and what this whole codebase is meant to fix.

The fix is an exception handler that records the reason, registered alongside the middleware:

async def http_exception_log_handler(request, exc):
 log.log(
 logging.ERROR if exc.status_code >= 500 else logging.WARNING,
 "request rejected",
 extra={
 "event": "request_rejected",
 "status_code": exc.status_code,
 "error_detail": _detail_text(exc.detail),
 },
 )
 return await _default_http_exception_handler(request, exc)

app.add_exception_handler(StarletteHTTPException, http_exception_log_handler)

It runs inside the middleware’s context, so the request, session, and user fields are already bound. It delegates to Starlette’s default handler, so the response the client receives is unchanged. And because detail is often a dict for structured errors, it gets JSON-serialized rather than str()-ed, so it stays greppable instead of turning into Python repr with single quotes.

Health probe overload

A load balancer probes /health every few seconds from every node, which means that those log lines can dominate log volume while reporting only that nothing happened, especially if they’re occuring frequently. I chose to suppress them, while still retaining useful information in the case that they fail:

Skip the request_start/request_end pair in the middleware for passing probes.
Drop uvicorn’s own access-log line for the same requests, via a filter on the root handler.

Uvicorn logs independently of your middleware, so leaving that half in place means you’ve halved the noise and kept the volume. A probe returning 4xx or 5xx is always logged – that’s the whole point of building this logging functionality. The correlation headers are still echoed on suppressed requests too, so if a probe starts failing and something retries, the retry is traceable.

Auto-capturing tracebacks

A log call made from inside an except block captures the active exception automatically, even when the caller writes log.warning("call failed") with no exc_info=True. I typically write logger.warning in error handlers constantly, and almost never remember exc_info. This results in a log full of “call failed” with no stacktrace. Capturing it by default means the traceback is there whether or not the author thought about it.

The record then carries both locations, which are usually different:

{
 "timestamp": "2026-05-26T19:09:37.015695Z",
 "level": "WARNING",
 "service": "lims-adapter",
 "message": "LIMS call failed: SQL query failed (upstream_status=405)",
 "func": "_lims_error",
 "filename": "errors.py",
 "lineno": 47,
 "request_id": "req-474c2493-18ea-42a0-be0f-5ce4d096b14c",
 "session_id": "ses-a55813da-3ab1-4471-9761-59d2ef988179",
 "user_id": "user@example.com",
 "path": "/samples/S-20260129-269/molecule",
 "exception": "Traceback (most recent call last):\n File \".../molecule.py\", line 10, in get_molecule\n ...\nLimsError: SQL query failed"
}

filename and lineno point at the error-handling code: the place someone decided to log. The traceback points somewhere else entirely, at the origin of the raise. That is where the bug lives. You want both, and they are rarely in the same file.

Field names and markers

Constants instead of string literals. A typo in fields.REQUEST_ID is a NameError at import. A typo in "request_id" is a silent schema divergence that nobody notices until a dashboard query quietly returns fewer rows than it should.

from logging_lib import fields

with log_context(**{fields.REQUEST_ID: rid, fields.USER_ID: uid}):
 log.info("handling request", extra={fields.EVENT: "request_start"})

Markers for categories. A top-level marker field tags a record for dashboards and alerting: AUDIT for user-initiated state changes, PERF for timings, SECURITY for auth and access control, SYSTEM for lifecycle, PIPELINE for job execution. It’s a small closed vocabulary, which is the point, and it makes queries trivial in whatever you’re storing logs in:

# CloudWatch Logs Insights
filter marker = "AUDIT"

# Loki
{service="lims-adapter"} | json | marker = "AUDIT"

What I now get

With the IDs in place, three questions become queries rather than investigations.

Sequencing. Filter on one request_id, sort by timestamp, and I have the ordered list of everything every service did for that one call, including the services the user never touched directly.

Lineage. requesting_service on each hop gives the call graph as it actually ran

Latency attribution. Every service records duration_s for the same request_id. The dashboard reports 3.1 seconds, service A reports 2.9, service B reports 2.7, and the LIMS query reports 2.6. The slow thing is the LIMS query, and it’s established without adding a single timer.

Service-to-Service Auth with Cognito Scopes

Tue, 09 Dec 2025 09:00:00 -0800

I’m building tools that help scientists run their day-to-day work. I work in a heavily regulated environment, subject to both U.S. and European digital policy requirements. The scenario we face regularly is: two services talk to each other. Then three. A dashboard calls a backend API, that API calls another one, a nightly job calls all of them. Somewhow, we have to answer how one service proves to another that it is allowed to make the call.

The first way I addressed this was to use Internal Bearer Tokens. Basically, you generate a long random string, store it in the AWS Secrets Manager, inject it into every service as an environment variable, have each service check inbound requests for it.

I replaced that solution with Cognito’s OAuth2 client-credentials flow. Here each service carries its own identity and each endpoint declares the scope it requires. We didn’t need to define any new infrastructure for this, since Cognito was already authenticating browser users through the ALB, so the service-to-service (S2S) path ended up reusing the same user pool, the same JWKS endpoint, and the same validation code that was already there.

Why a shared bearer token was not viable

The bearer token approach was really easy, but it wasn’t a viable approach in the long run, for a few reasons:

No expiry. A static token has no TTL. If it leaks through a log line, a memory dump, or a compromised container, it stays valid until you manually rotate it and run a redeploy of every service holding it. There is no safe window during rotation where both the old and new values work, unless you write that special-casing yourself.

No identity. When service B receives the token, it can verify the value matches what it was given, but it can’t tell who sent it. Every caller is indistinguishable, so you can only observe that the token was used.

No scope. The receiving service has no way to enforce what the caller may do. Either it accepts the token and grants full access, or it rejects it. There’s no way to express “service A may read but not write”.

Shared blast radius. The same secret works everywhere. One compromised service exposes the key to every other service, so a single breach leaves a huge hole.

Comparing these two solutions against each other, we have:

Property	Static bearer token	Cognito client credentials
Expiry	Never	~1 hour, auto-refreshed
Identity	None (opaque string)	Signed JWT with a `client_id` claim
Scope enforcement	Impossible	Per-endpoint, per-caller
Compromise blast radius	All services, indefinitely	One service, for TTL
Rotation	Manual, coordinated redeploy	Automatic
Auditability	“the token was used”	“service A called endpoint X”

Being able to attribute traffic to a caller enhances the utility of logs, which is most of the subject of my next post.

Two gates

There are two authentication and authorization gates in this architecture:

Gate 1 is the ALB. Everything in our system is either accesible from outside our VPC via the ALB, or has ingress/egress restricted to within the VPC. The ALB decides only how a request gets in: either this is an authenticated browser user who must complete an interactive Cognito login, or it’s a caller to hand straight to the service. The ALB knows nothing about your application’s permissions.

Gate 2 is the service. It validates identity and enforces scope. This is the real authorization check for S2S communication. It behaves identically no matter what language or host the caller runs on.

An incoming request has to clear both. A browser user who hasn’t been authenticated gets a 302 status to the Cognito login page via Gate 1 and never reaches the dashboard or API. A service with a valid token but the wrong scope passes Gate 1 but receives a 403 at Gate 2.

caller ──▶ [ Gate 1: ALB ] ──▶ [ Gate 2: service ] ──▶ your handler
 browser or machine? who are you, and what may you do?
 │ browser, │ no or invalid credential
 │ not logged in │ valid token, wrong scope
 ▼ ▼
 302 → Cognito login 401 / 403

What services can call what

Below is a diagram of allowed communication. Solid arrows are requests, and each one is labeled with the scope it must present. Dashed arrows are token issuance.

graph LR
Human(Browser user)
Robot(Laptop / CI job)
ALB{ALB}
Dash[Dashboard]
SvcA[Service A]
SvcB[Service B]
SvcC[Service C]
Cognito(Cognito)
Human -->|"session cookie, no scope"| ALB
Robot -->|"Authorization: Bearer"| ALB
ALB -->|"x-amzn-oidc-identity"| Dash
ALB -->|"Bearer passthrough"| SvcA
Dash -->|"service-a/read"| SvcA
Dash -->|"service-b/read"| SvcB
SvcA -->|"service-b/read"| SvcB
SvcA -->|"service-c/write"| SvcC
Cognito -.->|"token: service-a/read, service-b/read"| Dash
Cognito -.->|"token: service-b/read, service-c/write"| SvcA
classDef human fill:#e8f0fe,stroke:#4285f4
classDef machine fill:#e6f4ea,stroke:#34a853
class Human human
class Robot machine

A browser user reaches the dashboard with no scope at all, because the ALB already authenticated them interactively and the dashboard is the thing they’re allowed to look at. The dashboard’s own token is narrower than service A’s: it can read service A and service B, but nothing grants it write access to service C. If the dashboard is compromised, the write path isn’t reachable from there.

The permissions are just a table, so adding a consumer means adding a row, not changing any service’s code:

App client (the caller)	Allowed scopes
`dashboard`	`service-a/read`, `service-b/read`
`service-a`	`service-b/read`, `service-c/write`
`ci-runner`	`service-a/read`
`developer`	`service-a/read`, `service-b/read`

Cognito resource servers, custom scopes, and app clients

The resource server defines capabilities

A resource server represents a service that exposes capabilities. It has two name-like fields that do very different jobs:

# terraform code

resource "aws_cognito_resource_server" "service_b" {
 user_pool_id = var.cognito_user_pool_id

 # The scope prefix. Opaque string, conventionally a URL. Immutable in practice.
 identifier = "https://api.example.com/service-b"

 # Console display name only. Nothing references it.
 name = "service-b"

 scope {
 scope_name = "read"
 scope_description = "Read access to service-b"
 }

 scope {
 scope_name = "write"
 scope_description = "Write access to service-b"
 }

 scope {
 scope_name = "admin"
 scope_description = "Administrative access to service-b"
 }
}

The identifier field becomes the prefix of every scope the resource server defines, so the three scopes above are really:

https://api.example.com/service-b/read
https://api.example.com/service-b/write
https://api.example.com/service-b/admin

That full string is what a client requests, what appears in the JWT’s scope claim, and what a service compares against. name is a display label in the console and is referenced by nothing.

Only custom scopes work for machine-to-machine

The client-credentials flow works only with custom scopes from a resource server. I defined these scope manually in my FastAPI applications, and decorated the relevant endpoints with their respective scopes. The built-in OpenID scopes (openid, email, profile, aws.cognito.signin.user.admin) are for user-facing flows, and asking for one with grant_type=client_credentials fails. Every machine caller needs at least one resource server to exist, even if the service it calls has exactly one capability.

The app client also needs a secret and the flow explicitly enabled. A client without generate_secret cannot do client credentials at all, and the user pool needs a domain configured, since the token endpoint lives at https://<domain>.auth.<region>.amazoncognito.com/oauth2/token.

The app client selects a subset

An app client represents a caller of a service. One per calling service, with allowed_oauth_scopes listing every downstream capability it may request:

resource "aws_cognito_user_pool_client" "service_a_m2m" {
 name = "service-a-m2m"
 user_pool_id = var.cognito_user_pool_id

 generate_secret = true
 allowed_oauth_flows_user_pool_client = true
 allowed_oauth_flows = ["client_credentials"]

 # Full scope strings, exactly as composed above.
 allowed_oauth_scopes = [
 "https://api.example.com/service-b/read",
 "https://api.example.com/service-c/write",
 ]

 # Cognito validates that each scope exists, so the resource servers
 # must be created first. Terraform will not infer this ordering.
 depends_on = [
 aws_cognito_resource_server.service_b,
 aws_cognito_resource_server.service_c,
 ]
}

Service A is granted read on service B, and write on service C. It cannot write to service B even though that scope exists, because its app client never lists it as a viable permissions. Cognito rejects an app client referencing a scope that doesn’t exist yet, and Terraform sees no dependency between the two resources because the scope is a hand-written string rather than a reference to the resource server’s attributes.

Scopes model capabilities, not consumers. Service B declares that reading, writing, and administering are things that can be done to it (this is defined both in the FastAPI code via decorated endpoints, and in the Cognito resource server). But Service B doesn’t say anything about who can take those actions. Consumers get assigned action permissions through their app client’s allowed-scopes list.

That separation lets you onboard a caller without touching the service being called. A new consumer is one app client and one secret. Service B does not redeploy, does not learn the new caller’s name, and does not grow a config entry. Its code already says require_scope("read") and will keep saying that no matter how many services eventually call it.

Two possible gotchas

An omitted scope parameter is not the same as no scopes. If a token request leaves scope off entirely, Cognito issues a token carrying every scope the app client is allowed. This means that every downstream call then presents maximum privilege. Always request scopes explicitly, one per downstream call. This is also why caching tokens per scope rather than per client matters.

Removing a scope is harder than adding one. Cognito will not let you delete a scope from a resource server while an app client still lists it in allowed_oauth_scopes. The order is: remove it from every app client, apply, then remove it from the resource server. In Terraform that’s two applies, and doing it in one produces an error that names the resource server rather than the app client that’s actually the problem.

Validating a token

The receiving side has to check four things:

The signature, against Cognito’s public JWKS for the pool.
The issuer, so a token from some other pool isn’t accepted.
token_use == "access", so an ID token can’t be substituted for an access token.
The scope claim contains the specific scope the endpoint requires.

def _decode_token(self, token: str) -> dict:
 signing_key = self._jwk_client.get_signing_key_from_jwt(token)

 claims = jwt.decode(
 token,
 signing_key.key,
 algorithms=["RS256"],
 issuer=self.issuer,
 # Client-credentials tokens carry `client_id` instead of `aud`,
 # so skip audience verification and check token_use instead.
 options={"verify_aud": False},
 )

 if claims.get("token_use") != "access":
 raise HTTPException(401, "wrong token type")

 return claims

Client-credentials tokens have no aud claim. Instead, they carry client_id , so leaving audience verification on rejects every valid S2S token with a confusing error. Turning it off here is valid, but only because token_use and the issuer are being checked instead.

Scope enforcement then becomes a dependency on the router:

full_scope = f"{self.resource_server_id}/{self._scope}"
if full_scope not in claims.get("scope", "").split():
 raise HTTPException(403, f"token missing required scope: {full_scope}")

app = FastAPI(dependencies=[Depends(auth)]) # authenticated everywhere
router = APIRouter(dependencies=[Depends(auth.require_scope("read"))]) # plus a scope

There is also a design decision buried here. Browser requests arriving with ALB OIDC headers pass the scope check without a scope, since the ALB already authenticated the human interactively and browser sessions have no scopes to check.

Fetching a token

The outbound half fetches tokens from Cognito and caches them until they expire.

Cache per scope, not per client. A service calling three downstream APIs holds three tokens, each carrying only the access it needs for that call.

Guard the refresh with a lock, and re-check inside it. Under concurrent load, several coroutines will notice the expired token at the same moment and all stampede Cognito’s token endpoint:

async def get_token(self, scope: str) -> str:
 cached = self._tokens.get(scope)
 if cached and time.time() < cached.expires_at:
 return cached.access_token

 if scope not in self._locks:
 self._locks[scope] = asyncio.Lock()

 async with self._locks[scope]:
 # Re-check inside the lock: another coroutine may have fetched already.
 cached = self._tokens.get(scope)
 if cached and time.time() < cached.expires_at:
 return cached.access_token
 self._tokens[scope] = await self._fetch_new_token(scope)
 return self._tokens[scope].access_token

Expiry is stored with a 60 second skew subtracted, so a token is treated as expired slightly before Cognito thinks so. Without that, a token that passes the check and then spends 400ms in flight can arrive already invalid.

Required ALB rules

If your ALB has a Cognito authentication action on its listener rule, it applies to every request on that rule. A service presenting a valid Bearer token gets a 302 redirect to the Cognito login page. It never reaches your service, so all that careful token validation never runs.

The listener needs to rules, defined in order:

Higher priority: match requests carrying an Authorization: Bearer header and forward them straight to the EC2 target group, with no authenticate action.
Lower priority: everything else gets authenticate-cognito and the normal browser login.

With respect to Gate 1, you could also consider matching on source IP instead, forwarding anything from an allowlisted CIDR. This fixes Gate 1, and Gate 2 is unchanged either way.

However, a source-IP matching rule forwards without running the authenticate action. So once you allowlist the office or VPN range, a person opening a dashboard in a browser from that network skips the login entirely and arrives with no session and no token. This is DEFINITELY NOT what you want, because we want all requests to be authenticated. This is specific to browser users, and S2S communication still works.

Also, allowlisting the VPC’s own CIDR admits in-VPC services only. A public ALB sees a laptop’s corporate or VPN egress IP. That address is nowhere near the VPC range. So a VPC-CIDR rule never admits laptops, even on VPN. If in-VPC calls work and laptops get redirected to a login page, this is why.

Gotcha! Header bypass can worsen an authentication bug

We trust browser requests based on the presence of the x-amzn-oidc-identity header, and that the browser path skips scope checks. That’s normally safe, because the ALB’s Cognito action overwrites that header on every request it processes, so a client can’t forge it.

A bypassed request doesn’t run that action, so with a header-match rule, anyone on the internet can send:

Authorization: Bearer anything
x-amzn-oidc-identity: someone@example.com

The first header triggers the bypass. The second makes your service treat the request as an authenticated browser user, which skips the scope check entirely. The Bearer value is never validated, because the ALB-header path returns before the token path is reached.

The mitigations, either of which closes it:

Strip inbound x-amzn-oidc-* headers on the bypass path, so only the ALB can ever set them.
Dont trusting identity by presence of the header alone. Verify the signed x-amzn-oidc-data JWT rather than reading x-amzn-oidc-identity as a plain string.

The second is the better fix because it removes a whole class of bugs instance of one instance of it.

Calling it from anywhere

Something that confused me initially was that curl requests from the CLI would fail. But this was because we weren’t passing in any form of authentication into the request, which made local testing or scenarios like calling a service directly from an EC2 instance, fail. I enabled this functionality with two HTTP steps that are pretty much identical in Python, Java, or a shell script:

TOKEN=$(curl -s -X POST "https://<domain>.auth.<region>.amazoncognito.com/oauth2/token" \
 -u "$CLIENT_ID:$CLIENT_SECRET" \
 -d grant_type=client_credentials \
 -d 'scope=https://api.example.com/service-a/read' | jq -r .access_token)

curl "https://api.example.com/service-a/things" -H "Authorization: Bearer $TOKEN"

A Python client is a convenience wrapper around these steps – it’s not the ONLY way to run this authorization flow. Anything that can make an HTTPS request can participate. But we might want to alleviate this by building in a seperate app client for developers and for automation purposes.

Wrapping up

This was a pretty approachable problem, especially since we already had the infrastructure in place. We were already using Cognito to authenticate browser users through the ALB, and the client-credentials flow runs against the same user pool, the same JWKS endpoint, and the same validation code. We just needed one more OAuth flow on the system already in place.

Deploying Dagster to AWS ECS, Part 2: Pipelines

Wed, 05 Nov 2025 09:00:00 -0800

Here is the second of two posts on running Dagster in ECS. The first dealt with the platform side: the Daemon, the two Webservers, the ALB and Cognito wiring, the three security groups, the Cloud Map namespace, and the IAM that lets the Daemon launch anything at all.

The first component I referred to as dagster-platform. This one is about what I’m referring to as dagster-pipeline, the module you instantiate once per pipeline. Platform infrastructure gets deployed infrequently. Pipelines get deployed several times a day by whoever happens to be working on them. The reason I separated the two is that pushing a new job should never touch a security group, an IAM policy, or a load balancer rule.

The Contract

Part One ended with the four things the platform expects from a pipeline:

Register a service in the Cloud Map namespace, so the Daemon and Webserver can resolve it by DNS on port 4000.
Attach the usercode service to the platform’s usercode security group, so that traffic is actually allowed.
Tag its run roles with dagster:component and dagster:managed-by, so the Daemon is permitted to pass them to ECS.
Get its code location into workspace.yaml in SSM, so the Webserver and Daemon know it exists.

The platform defines the security group and the pipeline looks it up by name:

# dagster-pipeline/ecs.tf
data "aws_security_group" "usercode" {
 name = "${var.usercode_sg}-${var.platform_env}"
}

Here is the whole module and everything it touches. Blue is the platform, which this module does not create and only attaches to. Green is what the pipeline module builds.

graph LR
Daemon[daemon]
Web[webserver]
UcSvc[usercode service]
RunTD(run task def)
Run[run task]
CM(Cloud Map)
ECR(ECR)
SM(Secrets Manager)
PG(Postgres)
S3(S3)
SrcDB(source DB)
Web -->|4000| UcSvc
Daemon -->|4000| UcSvc
UcSvc -.->|register| CM
CM -.->|resolve| Daemon
UcSvc -.->|names| RunTD
Daemon -->|RunTask| RunTD
RunTD -.->|instantiates| Run
ECR -.->|image| UcSvc
ECR -.->|image| Run
SM -->|exec role| UcSvc
SM -->|task role| Run
UcSvc -->|5432| PG
Run -->|5432| PG
Run -->|443| S3
Run -->|query| SrcDB
classDef platform fill:#e8f0fe,stroke:#4285f4
classDef pipeline fill:#e6f4ea,stroke:#34a853
classDef ephemeral fill:#fef7e0,stroke:#f9ab00
classDef backend fill:#f1f3f4,stroke:#9aa0a6
class Daemon,Web platform
class UcSvc,CM,RunTD pipeline
class Run ephemeral
class ECR,SM,PG,S3,SrcDB backend

The run task definition is a definition, not a running thing: the usercode container names it in an environment variable, and the daemon is what actually instantiates it. The arrows pointing into the Secrets Manager land on different roles, because the usercode service has its credentials injected by ECS before it starts while the run task reads them itself at runtime.

The Two-Task-Definition Pattern

Each pipeline module creates two task definitions:

usercode task definition: the always-on code server. This runs dagster code-server start and stays alive, serving job definitions, sensors, schedules, and so on to the Daemon. It registers with Cloud Map so the platform can discover it.

run task definition: the task that executes a job. When a run is triggered, the EcsRunLauncher spins up a new Fargate task using this definition. It’s ephemeral, meaning it starts, runs the job, and exits.

The two definitions point at the same image and differ mainly in CPU/memory and in which IAM roles they use. The run definition’s command doesn’t really matter, since the launcher overrides it with the actual run command, but I set it to the same code-server command so that starting the task by hand does something reasonable. The usercode container needs to know about the run task definition so it can tell Dagster which task to launch, which can be achieved with two environment variables.

locals {
 dagster_current_image_env = [
 { name = "DAGSTER_CURRENT_IMAGE", value = data.aws_ecr_image.usercode.image_uri }
 ]

 dagster_container_context = {
 ecs = {
 task_definition_arn = aws_ecs_task_definition.run.arn
 container_name = "run"
 }
 }

 dagster_container_context_env = [
 { name = "DAGSTER_CONTAINER_CONTEXT", value = jsonencode(local.dagster_container_context) }
 ]
}

DAGSTER_CURRENT_IMAGE is the pipeline’s ECR image URI. DAGSTER_CONTAINER_CONTEXT tells Dagster which ECS task definition and container name to use for runs. Note that container_name has to match the name field of the container in the run task definition exactly. If it doesn’t, the run task launches and Dagster can’t find the container it’s supposed to be watching. Both of these go on the usercode container only, not the run container. Building DAGSTER_CONTAINER_CONTEXT from aws_ecs_task_definition.run.arn also creates the dependency that forces Terraform to build the run task definition first.

Registering With Cloud Map

The platform created the private DNS namespace, and each pipeline creates its own service record inside of that DNS namespace:

# dagster-pipeline/cloud_map.tf

# Look up the namespace created by the platform module
data "aws_service_discovery_dns_namespace" "dagster" {
 type = "DNS_PRIVATE"
 name = "pipelines-${var.env}.usercode"
}

# Each pipeline gets its own service record in the shared namespace
resource "aws_service_discovery_service" "usercode" {
 name = "${var.pipeline_name}-pipeline-${var.env}"

 dns_config {
 namespace_id = data.aws_service_discovery_dns_namespace.dagster.id

 dns_records {
 ttl = 300
 type = "A"
 }

 routing_policy = "MULTIVALUE"
 }
}

The ECS service ties itself to that record with a service_registries block, and from then on ECS registers the Fargate task’s private IP with Cloud Map automatically:

resource "aws_ecs_service" "usercode" {
 # ...
 service_registries {
 registry_arn = aws_service_discovery_service.usercode.arn
 }
}

The DNS name becomes <pipeline-name>-pipeline-<env>.pipelines-<env>.usercode, which goes into workspace.yaml:

# workspace.yaml (stored in SSM)
load_from:
 - grpc_server:
 host: ingest-pipeline-prod.pipelines-prod.usercode
 port: 4000
 location_name: "ingest"
 - grpc_server:
 host: transforms-pipeline-prod.pipelines-prod.usercode
 port: 4000
 location_name: "transforms"

“ingest” and “transforms” are what you see in the Dagster webserver when looking at each of your pipeline deployments.

TTL and Task Replacement

The ttl = 300 in that dns_records is important here. Fargate tasks get a new private IP every time they’re replaced, and a code server is replaced on every deploy. Cloud Map updates the A record immediately, but the Daemon and Webserver are resolving that name through the VPC resolver, which respects the TTL. For a few minutes after a deploy, they can be holding the IP of a task that no longer exists, and in practice, that means that a code location goes red (a.k.a looks like a failed deployment in the UI) in the UI right after a deploy and then fixes itself a few minutes later. Decreasing the TTL to 15 or 30 seconds makes deploys settle much faster, at the cost of more DNS queries, which for a handful of code servers is probably acceptable.

routing_policy = "MULTIVALUE" returns every healthy instance registered under the name. With desired_count = 1 there’s only ever one, but it’s the right policy if you later run more than one replica of a code server. Similarly, when you add a new pipeline, you need to update workspace.yaml in SSM and restart the Daemon and Webserver services so they pick up the new entry. I’m currently running this manually, but it’s worth automating in the future (but not a deal-breaker, just not something I’d consider “complete”).

Four Roles Per Pipeline

Each pipeline module creates four IAM roles: execution and task roles for both the usercode service (always-on code server) and the run task (ephemeral, launched per job execution).

Just like I mentioned in Part One, if you’re handling AWS secrets, if the value is injected by the ECS secrets block in the task definition, the execution role needs to read it. If your application code calls boto3 to fetch it, the task role needs to read it.

The roles are structurally identical, but the tags differ. The tags are what the platform’s iam:PassRole condition checks against:

# dagster-pipeline/iam.tf

locals {
 usercode_role_tags = {
 "dagster:component" = "usercode"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
 }

 run_role_tags = {
 "dagster:component" = "run"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
 }
}

# Run execution role, used by ECS to start the ephemeral run container
resource "aws_iam_role" "run_exec" {
 name = "${var.name_prefix}-${var.pipeline_name}-run-exec-role-${var.pipeline_env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
 tags = merge(local.run_role_tags, var.tags)
}

resource "aws_iam_role_policy_attachment" "run_exec_attach" {
 role = aws_iam_role.run_exec.name
 policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Run task role, used by the running job container to call AWS APIs
resource "aws_iam_role" "run_task" {
 name = "${var.name_prefix}-${var.pipeline_name}-run-task-role-${var.pipeline_env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
 tags = merge(local.run_role_tags, var.tags)
}

The Empty Resource List Trap

IAM rejects a policy statement with an empty Resource list, so a pipeline that connects to no databases can’t just get a policy document with zero ARNs in it. The document has to not exist at all:

data "aws_iam_policy_document" "db_secrets_read" {
 # A pipeline that reads no databases must not get this policy at all.
 count = length(var.db_secret_ids) > 0 ? 1 : 0

 statement {
 sid = "ReadDatabaseSecrets"
 effect = "Allow"
 actions = ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"]
 resources = [for s in data.aws_secretsmanager_secret.db_secrets : s.arn]
 }
}

resource "aws_iam_role_policy" "run_db_secrets" {
 count = length(var.db_secret_ids) > 0 ? 1 : 0
 name = "${var.name_prefix}-${var.pipeline_name}-run-db-secrets-${var.pipeline_env}"
 role = aws_iam_role.run_task.id
 policy = data.aws_iam_policy_document.db_secrets_read[0].json
}

Secrets Management

I’m using two different categories of secrets. Dagster’s own Postgres credentials are stored as a single Secrets Manager secret with JSON keys. These get injected as individual environment variables using the JSON key syntax in the ECS task definition secrets block:

locals {
 dagster_container_secrets = [
 {
 name = "DAGSTER_POSTGRES_HOSTNAME"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:hostname::"
 },
 {
 name = "DAGSTER_POSTGRES_USER"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:username::"
 },
 {
 name = "DAGSTER_POSTGRES_PASSWORD"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:password::"
 },
 {
 name = "DAGSTER_POSTGRES_DB"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:name::"
 }
 ]
}

The :key:: suffix trailing fields are the version stage and version ID. When those are empty, you get the most recent / current version. Since these are injected by ECS, the execution role is what needs GetSecretValue on the secret.

Source database credentials (for pipelines that read from other databases) are not injected at all anymore. My first version of the Terraform module wrote a .pgpass file at container startup: each secret ID came in as PGPASS_SECRET_JSON_1, PGPASS_SECRET_JSON_2 and so on, plus a PGPASS_SECRET_JSON_VARS variable listing which env vars to look at, and the entrypoint fetched each one and assembled the file. It worked, but it was messy and became really annoying to manage since I was dragging around another script for building the .pgpass file. It also meant credentials sat in the container environment and on disk for the life of the task, the entrypoint had to grow a chunk of logic that had nothing to do with running jobs, so I replaced it with the following

variable "db_secret_ids" {
 type = list(string)
 description = "Secrets Manager secrets for the databases this pipeline connects to. The task role is granted GetSecretValue on them; the application reads them at runtime."
 default = []
}

import json
import boto3


def get_connection_details(secret_id: str) -> dict:
 client = boto3.client("secretsmanager")
 secret = client.get_secret_value(SecretId=secret_id)
 return json.loads(secret["SecretString"])

The credentials are never in the environment and never on disk in the task, and adding a database is one more entry in a list variable.

What I’d Do (and still might do…) Differently

Automate workspace.yaml updates. Right now, adding a pipeline requires a manual SSM update and service restart. A Lambda triggered by ECS task state changes, or even a simple CI step, could handle this.

Use Cognito + proper OIDC federation for access control. The dual-webserver approach works, but it’s not pretty. If you control your IdP, set up Cognito federation from the start. I tried setting up a proxy server to redirect traffic based on users belonging to one Cognito user-group or another, but this ended up being a huge headache.

Consider ECS Exec for debugging. I didn’t enable it initially and spent a lot of time reading logs trying to diagnose container startup issues. aws ecs execute-command is worth the extra IAM policy on the task role.

Don’t put dagster.yaml in the platform images. Storing it in SSM and injecting at runtime was the right call for the daemon and webserver. Rebuilding (EVERY) image every time you want to change a config is not the move. For pipeline images, where the code changes anyway, baking it in is fine and simpler.

Fetch application secrets at runtime instead of injecting them. See above. This is the change I’m happiest about.

Actually, One More Thing

As I was writing up this post, I went back to the IAM code to check that I’d described the tag conditions correctly, and found a bug.

Here’s the setup again. The platform’s iam:PassRole policy only allows the Daemon to pass roles tagged dagster:component = "pipeline". The pipeline module’s local says something else:

run_role_tags = {
 "dagster:component" = "run" # not "pipeline"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
}

"run", not "pipeline". By that reading the Daemon should never be able to pass these roles, and every run should fail. But runs work fine in both environments, so I had never noticed. The correct approach is in the IaC merge command

tags = merge(local.run_role_tags, var.tags)

merge lets later arguments win, so var.tags overrides the module’s own local. And every caller passes a tag block that happens to include exactly the key in question:

tags = {
 "env" = var.pipeline_env
 "owner" = "data-platform"
 "application" = "dagster"
 "dagster:pipeline" = var.pipeline_name
 "dagster:component" = "pipeline" # this is what makes PassRole work
 "dagster:managed-by" = "terraform"
}

So the roles do come out tagged with pipeline, and the security control does work, but only because of the argument order in a merge and a tag the caller happens to set. A new pipeline that passes a tags map without dagster:component would plan clean, apply clean, start its code server, show up in the UI, and then fail the first time someone launched a run, with an AccessDenied on PassRole and nothing in the Terraform to suggest why.

The fix is to stop letting callers override the tag the policy depends on:

tags = merge(var.tags, local.run_role_tags)

and change the platform condition to match `"run"`, which is the value the module actually controls.

Wrapping Up

Getting Dagster onto ECS was genuinely hard for me. The documentation covers the “in a perfect world” scenario, but not the real-world complexity of integrating with existing cloud infrastructure that you don’t partially (or entirely) control. There are plenty of “in a perfect world with a clean slate, here’s how you do it” walkthroughs, but very little in the way of justification or documentation of architecture choices. The platform/pipeline split in Terraform has been a huge win for me. I’ve templated our Dagster pipeline development platform and put together templated Dockerfiles and docker-compose.yaml files. Deploying a new pipeline is now a terraform apply with a handful of variables, no platform infrastructure touched, no service restarts required (except for the workspace.yaml update, which I’m working on).

Deploying Dagster to AWS ECS, Part 1: The Platform

Tue, 04 Nov 2025 09:00:00 -0800

Here at Just-Evotec Biologics, the Data Platform team uses Dagster as our orchestration platform of choice. Many of our services are deployed to AWS ECS as Fargate or EC2 tasks. Getting Dagster onto ECS took me an embarrassing number of hours, and most of those issues stemmed from a lack of documentation around getting Dagster pipelines up and running in the cloud while NOT using Dagster+ (their premium service offering). The official docs cover the happy, perfect world path, but beyond that, you’re left stitching together forum threads, GitHub issues, and a lot of trial and error. I figured I’d put together some notes on what I built, and how I built it.

The components are of this system are two-fold – the actual Python-based pipeline code, and the Terraform code I used to deploy that code to ECS. The Terraform part also comes in two pieces, because I’ve split the deployment down the middle. In this post, I’ll detail the platform half: the long-lived infrastructure everything else plugs into, such as the webserver and the daemon. Part two covers the pipeline module, what you’d instantiate per pipeline and then redeploy repeatedly as you make updates to the pipeline code.

If you came here because something in your deployment cannot reach something else, it might be helpful for you to skip straight to the networking section, covering security groups, ports, service discovery. But if not, read on…

Setup

Our team already had a multi-environment AWS setup, with VPCs, subnets, security groups, tunneling, and load balancers in place. The goal was to plug Dagster into that existing infrastructure as cleanly as possible, with proper separation of concerns. Again, as I mentioned in some other posts, I obviously can’t share our company code, so I’ll just describe some of the approaches I took.

I split the Terraform code into two distinct modules:

dagster-platform: the long-lived infrastructure. Daemon, Webserver(s), shared IAM roles, ALB listener rules, SSM parameters, Cloud Map namespace, and S3/RDS backends.
dagster-pipeline: one instantiation per pipeline. Each pipeline gets its own ECS task definitions, service, Cloud Map entry, IAM roles, and log groups. You call this module once per pipeline and it slots right into the platform.

This split makes sense from a frequency-of-interaction standpoint. The platform piece deploys infrequently, and doesn’t need to be modified much. Pipelines deploy constantly, sometimes multiple times a day, and I don’t want to touch platform infrastructure every time I push a new job.

Architecture overview

Here’s the high-level picture:

ALB (existing)
 ├── /pipelines/admin/* → Webserver (read-write) [Cognito-authenticated]
 └── /pipelines/* → Webserver (read-only) [Cognito-authenticated]

Supporting Services
 ├── RDS PostgreSQL (run storage, event log, schedule storage)
 ├── S3 (compute logs)
 ├── AWS Secrets Manager (DB credentials, application secrets)
 ├── SSM Parameter Store (dagster.yaml, workspace.yaml)
 └── Cloud Map (service discovery for usercode servers)

ECS Cluster (existing)
 ├── dagster-daemon (always-on Fargate service)
 ├── dagster-webserver-rw (always-on Fargate service)
 ├── dagster-webserver-ro (always-on Fargate service)
 └── pipeline-<name> (always-on usercode service, one per pipeline)

Everything above the pipeline-<name> line is the platform – that’s what this current post is about. The Daemon and Webserver containers share the same dagster.yaml and workspace.yaml, loaded at runtime from SSM. Pipelines register themselves via Cloud Map so the Daemon can find them.

The read-only webserver problem

Dagster’s dagster-webserver supports a --read-only flag, but there’s no built-in authentication layer that lets you say “scientists get read-only access, data engineers get admin access.” Dagster does have a subscription tier called Dagster+ that does offer this functionality, but we do not pay for that. So, we’re lefting putting our own solution in place. I’m not sure if there is a “right” answer here. One solution, given our constraints, is to combine Cognito with our IdP federation, but since I don’t control our company’s IdP tenant, I couldn’t get that working. Instead, I deployed two separate ECS services, one read-write and one read-only, each behind its own ALB target group and listener rule.

Read-write webserver command:

dagster-webserver -h 0.0.0.0 -p 3000 \
 -w /opt/dagster/dagster_home/workspace.yaml \
 -l /pipelines/admin

Read-only webserver command:

dagster-webserver -h 0.0.0.0 -p 3000 \
 -w /opt/dagster/dagster_home/workspace.yaml \
 -l /pipelines \
 --read-only

The -l flag sets the URL prefix. The ALB rules do the routing:

/pipelines/admin* → read-write target group (higher priority)
/pipelines* → read-only target group (lower priority)

Both rules are Cognito-authenticated, so anyone hitting either URL has to log in. Scientists get the read-only URL. Engineers get the admin URL. Simple, but it works.

Two actions per listener rule

An authenticated listener rule is two actions in order. The authenticate-cognito action runs first, then the forward action sends the request to the target group. If you only write the forward action, the rule works and nobody has to log in, which is the kind of mistake you don’t notice until someone tells you they never saw a login screen.

resource "aws_lb_listener_rule" "pipelines_admin_rw" {
 listener_arn = data.aws_alb_listener.webserver.arn
 priority = local.admin_listener_priority

 action {
 type = "authenticate-cognito"
 authenticate_cognito {
 user_pool_arn = data.aws_cognito_user_pool.dagster.arn
 user_pool_client_id = data.aws_cognito_user_pool_client.dagster.id
 user_pool_domain = data.aws_cognito_user_pool.dagster.domain

 on_unauthenticated_request = "authenticate"
 scope = "openid"
 session_timeout = 3600
 }
 }

 action {
 type = "forward"
 target_group_arn = aws_lb_target_group.webserver_rw.arn
 }

 condition {
 path_pattern {
 values = ["/${var.alb_rule_listener_rule_path}/admin*"]
 }
 }
}

The read-only rule is identical except for the path pattern and target group, plus a depends_on pointing at the admin rule so the two are never created in the wrong order.

Finding an open listener priority

ALB listener rule priorities have to be unique across the whole listener, and they need to be specified. Neither Terraform nor AWS will “infer” the next open value. Our ALB is shared with other applications, so hardcoding numbers means constantly checking what’s available in the console or through the CLI. I ended up writing a small bash script that the external data source calls at terraform plan time. It lists the existing rules and returns the first N unused priority slots:

data "external" "listener_priorities" {
 program = ["bash", "${path.module}/external/next_listener_priorities.sh"]

 query = {
 listener_arn = data.aws_alb_listener.webserver.arn
 region = data.aws_region.current.id
 profile = var.aws_profile
 count = 2
 }
}

locals {
 admin_listener_priority = tonumber(data.external.listener_priorities.result.priority_1)
 ro_listener_priority = tonumber(data.external.listener_priorities.result.priority_2)
}

And the script itself, which is a jq expression over describe-rules:

#!/usr/bin/env bash
set -euo pipefail

INPUT_JSON="$(cat)"
LISTENER_ARN="$(jq -r '.listener_arn' <<<"$INPUT_JSON")"
REGION="$(jq -r '.region' <<<"$INPUT_JSON")"
COUNT="$(jq -r '.count // 2' <<<"$INPUT_JSON")"

RULES_JSON="$(aws --region "$REGION" elbv2 describe-rules \
 --listener-arn "$LISTENER_ARN" --output json)"

# Find the first COUNT missing priorities in [1..50000]
jq -c --argjson count "$COUNT" '
 [ .Rules[].Priority | select(. != "default") | tonumber ] | unique as $used
 | reduce range(1; 50001) as $i
 ({out: []};
 if (.out | length) >= $count then .
 elif ($used | index($i)) == null then .out += [$i]
 else . end
 )
 | .out as $m
 | reduce range(0; ($m | length)) as $idx
 ({}; . + {("priority_" + (($idx + 1) | tostring)): ($m[$idx] | tostring)})
' <<<"$RULES_JSON"

The external data source runs on every plan, so the machine running Terraform needs the AWS CLI and jq installed and credentials that can call describe-rules (this could be your local machine, your CI/CD runner, an EC2 instance, etc.) The priorities are read at plan time, so if someone else claims a slot between your plan and your apply, the apply fails, though that’s rare enough that I’ve been willing to live with it. Basically, hardcoding priorities was not an option I wanted to entertain.

Health checks and ALB path prefixes

The ALB health checks for the webservers need to use the correct path prefix. If your webserver is mounted at /pipelines and your health check pings /, it’ll get a redirect (or a 404) and the target will never go healthy.

Read-only health check path: /<your-prefix> Read-write health check path: /<your-prefix>/admin

health_check {
 protocol = "HTTP"
 path = "/${var.alb_rule_listener_rule_path}/admin"
 matcher = "200-399"
 healthy_threshold = 2
 unhealthy_threshold = 3
 interval = 30
 timeout = 25
}

Set the matcher to 200-399 to handle redirects during startup. The Dagster webserver takes 15 to 30 seconds to come up on a cold start, so unhealthy_threshold and interval need to be generous enough that a slow container start doesn’t put the service into a restart loop.

dagster.yaml: the compute and storage specs

The dagster.yaml configuration basically let’s you define job queing, which system will run your jobs (here, ECS), where logs and compute history will be stored, etc. Here’s what our final config looks like (pretty similar to what’s already out there):

run_coordinator:
 module: dagster.core.run_coordinator
 class: QueuedRunCoordinator
 config:
 max_concurrent_runs: 15

scheduler:
 module: dagster.core.scheduler
 class: DagsterDaemonScheduler

run_launcher:
 module: dagster_aws.ecs
 class: EcsRunLauncher
 config:
 use_current_ecs_task_config: true
 include_sidecars: true
 run_task_kwargs:
 cluster: "your-ecs-cluster-name"

compute_logs:
 module: dagster_aws.s3.compute_log_manager
 class: S3ComputeLogManager
 config:
 bucket:
 env: DAGSTER_LOG_BUCKET
 prefix: "compute_logs"
 region: "your-aws-region"

run_storage:
 module: dagster_postgres.run_storage
 class: PostgresRunStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

schedule_storage:
 module: dagster_postgres.schedule_storage
 class: PostgresScheduleStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

event_log_storage:
 module: dagster_postgres.event_log
 class: PostgresEventLogStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

retention:
 schedule:
 purge_after_days: 90
 sensor:
 purge_after_days:
 skipped: 7
 failure: 30
 success: -1

The ECS task configuration was confusing to me initially. use_current_ecs_task_config: true tells the EcsRunLauncher to take the network configuration (subnets, security groups, cluster) from the task that requested the run rather than making you restate all of it in run_task_kwargs. Combined with a per-pipeline run task definition, which is covered in part two, it means each pipeline’s runs launch into the same networking as that pipeline’s code server, with that pipeline’s own IAM roles. You don’t need to define a shared task definition across pipelines, which keeps the principle of least privilege in play.

Every credential in that file is an env: reference, and nothing secret is in the file itself.

Where the platform config lives

There are two config files: dagster.yaml (instance config, above) and workspace.yaml (the list of code locations the Daemon and Webserver load over gRPC). Baking config into the Docker image means rebuilding and redeploying the image every time you tweak a config value. For the Daemon and Webserver that’s a bad idea, because those images almost never change, but the config does. So Terraform writes both files into SSM parameters:

resource "aws_ssm_parameter" "dagster_yaml" {
 name = "${var.ssm_prefix}/dagster.yaml"
 type = "String"
 value = var.dagster_yaml
}

resource "aws_ssm_parameter" "workspace_yaml" {
 name = "${var.ssm_prefix}/workspace-${var.platform_env}.yaml"
 type = "String"
 value = var.workspace_yaml
}

and the task definitions inject them as container secrets:

locals {
 dagster_config_ssm_secrets = [
 { name = "DAGSTER_YAML", valueFrom = aws_ssm_parameter.dagster_yaml.arn },
 { name = "WORKSPACE_YAML", valueFrom = aws_ssm_parameter.workspace_yaml.arn },
 ]

 common_platform_env = [
 { name = "DAGSTER_CONFIG_FROM_ENV", value = "1" },
 { name = "DAGSTER_DEPLOYMENT_ENV", value = var.platform_env },
 { name = "DAGSTER_HOME", value = var.dagster_home },
 { name = "DAGSTER_LOG_BUCKET", value = "${var.dagster_log_bucket}-${var.platform_env}" },
 ]
}

DAGSTER_CONFIG_FROM_ENV is a flag the entrypoint script checks. When it’s set, the script writes $DAGSTER_YAML and $WORKSPACE_YAML out to $DAGSTER_HOME/dagster.yaml and $DAGSTER_HOME/workspace.yaml before starting the process, and leaves whatever is in the image alone otherwise. Changing a config value is then an SSM update and a service restart, not a full image rebuild and redeployment.

The ECS secrets block works with plain SSM String parameters, not just SecureString and Secrets Manager, and standard SSM parameters are capped at 4KB of value, so a workspace.yaml with a lot of code locations will eventually need an advanced parameter (8KB) or a different mechanism. Pipeline containers do the opposite and ship their config in the image. That’s covered in Part Two. I also changed my mind about it after writing these posts, so I add some thoughts about that as well.

Networking: the part that took me the longest

I found the networking stuff to be the most confusing. Multiple features all have to be correct before anything communicates to anything else.

What talks to what

Before any of the Terraform, it was worthwhile to me to write the traffic down explicitly, because the number of distinct flows is small and each flow has to be allowed somewhere:

graph LR
Browser(Browser)
ALB{ALB}
RW[webserver-rw]
RO[webserver-ro]
Daemon[daemon]
Uc[usercode]
Run[run task]
CM(Cloud Map)
ECSAPI(ECS API)
PG(Postgres)
S3(S3)
Browser -->|443| ALB
ALB -->|"/admin*"| RW
ALB -->|"/*"| RO
RW -->|4000| Uc
RO -->|4000| Uc
Daemon -->|4000| Uc
Daemon -->|RunTask| ECSAPI
ECSAPI -->|launches| Run
Uc -.->|register| CM
CM -.->|resolve| Daemon
CM -.->|resolve| RW
RW -->|5432| PG
Daemon -->|5432| PG
Uc -->|5432| PG
Run -->|5432| PG
Run -->|443| S3
classDef platform fill:#e8f0fe,stroke:#4285f4
classDef pipeline fill:#e6f4ea,stroke:#34a853
classDef ephemeral fill:#fef7e0,stroke:#f9ab00
classDef backend fill:#f1f3f4,stroke:#9aa0a6
class RW,RO,Daemon platform
class Uc pipeline
class Run ephemeral
class CM,ECSAPI,PG,S3 backend

Blue is the platform module, green is a pipeline, amber is the ephemeral run task, grey is something AWS runs for you. Solid arrows are request traffic, labelled with the port. Dotted arrows are service discovery, which carries no application data. Here is the same thing as a table:

From	To	Port	Why
ALB	Webserver	3000 (`webserver_port`)	Serving the UI
Webserver	Usercode server	4000	Loading code locations over gRPC
Daemon	Usercode server	4000	Polling schedules and sensors
Daemon	ECS API	443	`RunTask` to launch runs
Daemon	EC2 API	443	`DescribeNetworkInterfaces` to resolve task IPs
Daemon, Webserver, Usercode, Run	RDS Postgres	5432	Run storage, event log, schedule storage
Daemon, Webserver, Usercode, Run	S3	443	Compute logs
All tasks (at startup)	ECR, Secrets Manager, SSM, CloudWatch Logs	443	Pulling the image, injecting secrets and config

The Daemon never receives inbound traffic from anything, so its security group needs no ingress rules at all.

Three security groups

The platform module creates all three security groups, including the one used by pipelines. The usercode security group uses security group references, not CIDR blocks, to restrict inbound port 4000 to only the Daemon and Webserver, so nothing else in the VPC can reach a code server. While a CIDR rule like 10.0.0.0/16 would work too, doing so means anything that gets a private IP in that range can open a gRPC connection to your code servers, and you don’t want that. A code server could tell any caller what jobs exist and let them be launched. Referencing the daemon and webserver security groups instead means the rule stays correct as subnets change and there’s no CIDR math to get wrong.

# dagster-platform/vpc.tf

# Daemon: outbound-only. It talks to ECS APIs and usercode servers, never receives inbound.
resource "aws_security_group" "daemon" {
 name = "${var.name_prefix}-daemon-sg-${var.env}"
 description = "Dagster Daemon. Outbound only."
 vpc_id = var.vpc_id

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# Webserver: inbound from ALB only, on the webserver port.
resource "aws_security_group" "webserver" {
 name = "${var.name_prefix}-webserver-sg-${var.env}"
 description = "Dagster webservers (RO+RW). Inbound from ALB only."
 vpc_id = var.vpc_id

 ingress {
 description = "Inbound from ALB security group"
 from_port = var.webserver_port
 to_port = var.webserver_port
 protocol = "tcp"
 security_groups = [var.alb_sg_id] # reference your existing ALB SG here
 }

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# Usercode servers: inbound TCP 4000 from daemon and webserver SGs only.
# This SG is shared across all pipeline usercode services.
resource "aws_security_group" "usercode" {
 name = "${var.name_prefix}-usercode-sg-${var.env}"
 description = "Dagster usercode. Allow TCP 4000 from daemon and webserver only."
 vpc_id = var.vpc_id

 ingress {
 description = "gRPC from daemon and webserver"
 from_port = 4000
 to_port = 4000
 protocol = "tcp"
 security_groups = [
 aws_security_group.daemon.id,
 aws_security_group.webserver.id,
 ]
 }

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

The usercode security group is defined here and looked up by name in each pipeline module, so the platform owns the definition and pipelines just join it.

How do tasks get their networking?

The Dagster documentation unfortunately does not describe this clearly. There is no security group anywhere in the Terraform for run tasks. So how do they run? As I mentioned above, the answer is use_current_ecs_task_config: true in dagster.yaml. When the EcsRunLauncher launches a run, it reads the network configuration of the task making the request (the usercode code server) and reuses it. Run tasks land in the same subnets and the same usercode security group as the code server that spawned them automatically. Meaning:

A run task inherits inbound 4000 from the daemon and webserver
A run task needs the same egress as the code server: Postgres to write events, S3 for compute logs, and everything in the startup list below. Egress is where run tasks actually fail, never ingress.
If you lock the usercode security group’s egress down, you’re locking down runs too, and the failure will look like a job that starts and then hangs rather than a network error.

Each pipeline’s runs get that pipeline’s networking and that pipeline’s IAM roles, with nothing shared and nothing restated. The pipeline’s tasks are inheriting the networking rules defined for that whole pipeline.

Private Subnets Need a Path to AWS APIs

All of these tasks run in private subnets with assign_public_ip = false. A Fargate task in a private subnet still has to reach several AWS endpoints over the AWS API surface before that container code runs at all: ECR to pull the image (both the API and the layer storage in S3), Secrets Manager and SSM to inject secrets and config, and CloudWatch Logs to write those outputs. That traffic needs either a NAT gateway in the route table or VPC interface endpoints for each of those services, plus an S3 gateway endpoint. Our VPC already had a NAT gateway, so I leveraged that. If that path is missing, the task doesn’t fail in a way that mentions networking. It sits in PROVISIONING or PENDING, then stops with a ResourceInitializationError about being unable to pull the image or retrieve a secret. Nothing in the container logs, because the container never started, and nothing in the Dagster UI, because Dagster never got far enough to know about it.

What Each Failure Actually Looks Like

This table contains rough debugging notes I wish I’d had:

Symptom	Usual cause
Task stuck in `PROVISIONING`, then `ResourceInitializationError`	No egress path from the private subnet. Missing NAT route or VPC endpoints
Code location shows as failed in the UI, gRPC deadline exceeded	Usercode SG isn’t allowing 4000 from the webserver SG
Schedules and sensors never tick, but the UI loads fine	Same rule, but the daemon SG. The webserver and daemon are separate sources
Code location resolves intermittently after a deploy	Cloud Map DNS TTL still serving the old task’s IP. See part two
ALB target never goes healthy	Health check path missing the URL prefix, or the matcher rejecting a redirect
Runs launch and immediately fail with no logs	IAM, not networking. Usually the missing `ecs:TagResource`
Run starts, then hangs without writing events	Egress from the usercode SG. Run tasks inherit it

Generally, if you’re seeing an error, I’d start with examining the IAM roles first.

Service Discovery: The Namespace

The Daemon needs to find each pipeline’s code server. On ECS Fargate, the standard approach is AWS Cloud Map (private DNS). The platform creates the namespace, and each pipeline creates its own service record within it.

# dagster-platform/cloud_map.tf

resource "aws_service_discovery_private_dns_namespace" "dagster" {
 name = "pipelines-${var.env}.usercode" # e.g. "pipelines-prod.usercode"
 description = "Private DNS namespace for Dagster usercode servers"
 vpc = var.vpc_id
}

Two requirements come with a private DNS namespace. It’s bound to exactly one VPC, so a multi-VPC setup needs a namespace per VPC (which is part of why the name carries the environment). And the VPC needs both enableDnsSupport and enableDnsHostnames turned on, or the records resolve from nothing and every code location fails to load with what looks like a connection error rather than a DNS one.

IAM: Probably the crux of everything

There are two IAM roles per service: an execution role (used by ECS to pull images, write logs, and fetch secrets before the container starts) and a task role (used by the running container to call AWS APIs at runtime). Regarding AWS secrets: if the value is injected by the ECS secrets block in the task definition, the execution role needs to read it, because something needs to inject it into the task definition. If your application code calls boto3 to fetch it, the task role needs to read it. Yes, I was confused initially too.

The Shared Trust Policy

All ECS task roles share the same trust policy:

data "aws_iam_policy_document" "ecs_task_assume_role" {
 statement {
 actions = ["sts:AssumeRole"]
 principals {
 type = "Service"
 identifiers = ["ecs-tasks.amazonaws.com"]
 }
 }
}

Execution Roles (Webserver + Daemon)

Execution roles need the standard ECS managed policy, plus access to Secrets Manager and SSM so ECS can inject those values before the container starts:

# dagster-platform/iam.tf

resource "aws_iam_role" "daemon_execution" {
 name = "${var.name_prefix}-daemon-exec-role-${var.env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

resource "aws_iam_role_policy_attachment" "daemon_exec_attach" {
 role = aws_iam_role.daemon_execution.name
 policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Execution role needs Secrets Manager access so ECS can inject secrets into the container
resource "aws_iam_role_policy" "daemon_exec_db_secret" {
 name = "${var.name_prefix}-daemon-exec-db-secret-${var.env}"
 role = aws_iam_role.daemon_execution.id
 policy = data.aws_iam_policy_document.dagster_db_secret_read.json
}

# Execution role needs SSM access to inject dagster.yaml and workspace.yaml as env vars
resource "aws_iam_role_policy" "daemon_exec_ssm" {
 name = "${var.name_prefix}-daemon-exec-ssm-${var.env}"
 role = aws_iam_role.daemon_execution.id
 policy = data.aws_iam_policy_document.dagster_ssm_read.json
}

The webserver execution role is identical in structure.

Daemon Task Role

The Daemon’s task role needs to launch, describe, stop, and tag ECS run tasks. Dagster tags every run task it spawns with run metadata, and if the permission is absent, runs will launch and then immediately fail.

resource "aws_iam_role" "daemon_task" {
 name = "${var.name_prefix}-daemon-task-role-${var.env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

# Allow launching run tasks for any task def matching "pipeline-*-run-*"
# Scoped to the specific cluster so the daemon can't escape to other clusters
data "aws_iam_policy_document" "ecs_run_tasks" {
 statement {
 sid = "RunDescribeStopRunTasks"
 effect = "Allow"
 actions = ["ecs:RunTask", "ecs:DescribeTasks", "ecs:StopTask"]
 resources = [
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task-definition/pipeline-*-run-*:*",
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task/${var.cluster_name}/*",
 ]
 condition {
 test = "ArnEquals"
 variable = "ecs:cluster"
 values = [data.aws_ecs_cluster.cluster.arn]
 }
 }

 # DescribeTaskDefinition has no resource-level support, must be "*"
 statement {
 sid = "DescribeTaskDefinitions"
 effect = "Allow"
 actions = ["ecs:DescribeTaskDefinition"]
 resources = ["*"]
 }
}

resource "aws_iam_role_policy" "daemon_ecs_run" {
 name = "${var.name_prefix}-daemon-ecs-run-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_run_tasks.json
}

# Allow tagging ECS tasks. Without this, Dagster run tasks fail silently at launch.
data "aws_iam_policy_document" "ecs_tag_resource" {
 statement {
 sid = "AllowTagEcsTasks"
 effect = "Allow"
 actions = ["ecs:TagResource"]
 resources = [
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task/${var.cluster_name}/*"
 ]
 }
}

resource "aws_iam_role_policy" "daemon_ecs_tag" {
 name = "${var.name_prefix}-daemon-ecs-tag-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_tag_resource.json
}

# The daemon also needs ec2:DescribeNetworkInterfaces to resolve Fargate task IPs
resource "aws_iam_role_policy" "daemon_describe_enis" {
 name = "${var.name_prefix}-daemon-describe-enis-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ec2_describe_enis.json
}

PassRole Contract

The last piece of the daemon’s task role is the one that ties the two modules together. To launch a run task, the daemon has to pass that task’s execution and task roles to ECS, and THOSE roles live in the pipeline module, not here. The naive version grants iam:PassRole on *, which hands the daemon the ability to pass any role in the account to ECS, but that doesn’t follow the principle of least priviledge. Instead, I scoped it with tag conditions:

data "aws_iam_policy_document" "ecs_pass_dagster_roles" {
 statement {
 sid = "PassOnlyDagsterRunRoles"
 effect = "Allow"
 actions = ["iam:PassRole"]
 resources = ["*"] # resource-level not supported for PassRole; use tag conditions instead

 condition {
 test = "StringEquals"
 variable = "iam:PassedToService"
 values = ["ecs-tasks.amazonaws.com"]
 }
 # Only allow passing roles tagged as dagster pipeline components
 condition {
 test = "StringEquals"
 variable = "iam:ResourceTag/dagster:component"
 values = ["pipeline"]
 }
 condition {
 test = "StringEquals"
 variable = "iam:ResourceTag/dagster:managed-by"
 values = ["terraform"]
 }
 }
}

resource "aws_iam_role_policy" "daemon_pass_roles" {
 name = "${var.name_prefix}-daemon-pass-run-roles-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_pass_dagster_roles.json
}

iam:PassRole doesn’t support resource-level scoping the way you’d want, so "*" plus a strict tag condition is the practical answer. That means that you can add new pipelines without ever touching the platform’s IAM policies. As long as a new pipeline’s run roles carry the right tags, the daemon can pass them.

What the Platform Expects From a Pipeline

In actuality, the tag condition is really a contract. To plug into this Dagster platform, a pipeline has to:

Register a service in the Cloud Map namespace, so the Daemon and Webserver can resolve it by DNS on port 4000.
Attach its usercode service to the platform’s usercode security group, so that traffic is actually allowed.
Tag its run roles with dagster:component and dagster:managed-by, so the Daemon is permitted to pass them to ECS.
Get its code location into workspace.yaml in SSM, so the Webserver and Daemon know it exists.

Part two covers the module that does all four, plus the two-task-definition pattern, secrets, and how deploys actually roll a new image onto ECS.

Building a Schema Registry from Scratch for a Scientific Data Platform

Tue, 24 Jun 2025 09:00:00 -0700

When I joined Just-Evotec Biologics (first as a data scientist, now as a data platform engineer), I inherited a data ecosystem that probably looked pretty familiar to what lots of others have dealth with: a dozen scientific instruments each outputting data in proprietary formats, a LIMS system with its own schema, upstream and downstream experimental systems, Excel workbooks scientists had been maintaining for years but hidden away from any production storage system, and a swathe of applications that had been developed previously but had very little in the way of data governance. We didn’t have a central schema definitions, very little in the way of contracts between producers (e.g. instrumentation) and consumers (e.g. scientists, applications, clients, more instruments), and no way to ask or answer “what’s being pulled down here?”.

I’ve been slowly building up our data platform, including implementing a formal schema registry. Here, I’ll cover some core design decisions I made and patterns that worked for me for the registry. The implementation is domain (and in our case, functional group and stakeholder) agnostic.

Why not just use an existing registry tool?

Some alternative solutions to implementing / using a schema registry include Confluent’s Schema Registry, AWS Glue, Great Expectations, etc. Confluent’s Schema Registry is excellent for Kafka-based event streams, AWS Glue has a data catalog, and Great Expectations handles data quality well (just like Pandera). But none of those really addressed all or most of my needs, or fit in with my environment: a mix of validated DataFrames, LIMS API outputs, Parquet files in S3, and a team of scientists, some of whom wrote Python but weren’t data engineers. I wanted the following:

Define schemas as Python classes with static type annotations
Automatically serialize those classes to human-readable YAML
Query schemas by metadata (tier, data lineage stage, source system)
Reconstruct a live Pandera model from a stored YAML definition at runtime

We’re primarily working with data tables (as opposed to events), so I wanted a coupling between a type agnostic representation like YAML and Pandera. However, I’ve since extended a lot of this functionality for event-based schemas. At some point, I might want to revisit Glue again so I don’t reinvent the wheel.

Core architecture

The general schema development flow is: write Python → export YAML → publish to S3 → consume via the registry client at runtime.

author a Pandera contract (contracts/domains/**/schema.py)
 → export-models-to-yaml # Python model to canonical YAML
 → upload-yaml-to-s3 # publish, through the canonical-dtype gate
 S3 (what consumers actually read)
 → REGISTRY.get_pandera_model(name) # at runtime, in a pipeline
 → a live Pandera DataFrameModel

The important property of this diagram is the direction of the arrow into S3. Python is how schemas are authored, but S3 is what consumers read. A pipeline never imports a contract class. It fetches YAML and builds a Pandera model from it at runtime, which means a schema change propagates by re-publishing to S3, with no rebuild or redeploy of any consumer. I initially didn’t build this in, and found myself needed to re-install the registry every time I authored a new contract. Not ideal.

The repository is a monorepo of three independently versioned pieces:

Piece	What it is	Who uses it
the service	Contract definitions, templates, source-system specs, the FastAPI API, and the export/publish tooling	Deployed as a Lambda. Not installed as a library.
the client	Fetch schemas from S3 and deserialize them to Pandera models, plus the canonical dtype vocabulary	Pipelines, and anything validating data against a schema (e.g. dashboards)
the data IO package	Connectors for the LIMS and for the parquet/delta lakehouse	Pipelines

I started with one package and split it once pipelines started depending on it. A pipeline that just wants to validate a DataFrame should not be installing FastAPI, jsonschema, and the whole publish toolchain to do it. The client should be thin, and the dependency direction one-directional a.k.a. the service depends on the client.

Repository layout

The code tree looks like this. Names are genericized so as to hide our proprietary namespaces and functionality, but the shape is exactly what we run:

schema-registry/
├── registry/ # the service
│ ├── contracts/
│ │ ├── templates/ # JSON Schema per schema_type
│ │ │ ├── dataframe/v1_0/schema.yaml
│ │ │ ├── event/v1_0/schema.yaml
│ │ │ └── api/v1_0/schema.yaml
│ │ ├── domains/ # the contracts themselves
│ │ │ ├── lab/
│ │ │ │ └── analytics/
│ │ │ │ └── cell_count/
│ │ │ │ ├── v1_0/
│ │ │ │ │ ├── schema.py # authored
│ │ │ │ │ └── schema.yaml # generated, committed
│ │ │ │ └── v2_0/
│ │ │ │ ├── schema.py
│ │ │ │ └── schema.yaml
│ │ │ ├── operations/
│ │ │ └── enterprise/
│ │ └── utilities/dataframe/export.py # Python model → YAML
│ ├── integrations/
│ │ └── source_specs/ # contract → physical source table
│ │ └── lab/analytics/cell_count/v1_0/spec.yaml
│ ├── api/ # FastAPI app (deployed as a Lambda)
│ │ ├── app.py
│ │ ├── auth.py
│ │ └── routers/{domains,schemas,specs}.py
│ ├── publish/sync_schemas.py # validation gates + S3 upload
│ └── scripts/ # the two CLI entry points
│
└── packages/
 ├── schema-registry-client/ # what pipelines + dashboards install
 │ └── schema_registry_client/
 │ ├── contracts/utilities/dataframe/
 │ │ ├── dtypes.py # the canonical vocabulary
 │ │ ├── mixins.py # SchemaInfo, SchemaMeta, ContractMixin (e.g. constants + fields applied to all schemas)
 │ │ ├── registration.py # the decorator + path checks
 │ │ └── base.py # base DataFrameModel configs
 │ └── io/
 │ ├── registry.py # the consumer-facing Registry
 │ ├── backends/s3.py # index-backed S3 reads
 │ └── adapters/ # YAML → live Pandera model
 └── data-io/ # source-system and lakehouse connectors

I wanted to be able to seperate the client from the registry and contracts, so I want to emphasize which side of the packages/ line each concern lives on. Everything a consumer needs, the dtype vocabulary, the identity dataclasses, the deserializer, sits in the client. Everything about producing the registry, discovery, export, validation gates, upload, and the API, sits in the service. The decorator lives in the client rather than the service, which looks odd at first, but contracts are authored against it and the deserializer needs the same SchemaInfo shape when it rebuilds a model, so it belongs on the shared side.

The generated schema.yaml sitting next to the authored schema.py and getting committed is intentional. A schema change shows up in code review as a readable diff against the previous commit of the actual YAML contract, not just as a Python class that someone would have to mentally compile.

Adding a schema, end to end

The whole process for a contributor is only five steps:

Create contracts/domains/<domain>/<subdomains>/<entity>/v1_0/schema.py and write the Pandera model, decorated with @register_dataframe_schema.
If it maps to a physical source table, add the matching spec under integrations/source_specs/.
Run the export command. It walks the tree, checks that the declared name and version match the directory path, and writes schema.yaml next to each schema.py.
Commit both files. CI re-runs the export and fails the build if anything is out of date.
Run the upload command. It validates every document against its template, runs the canonical dtype gate, uploads to S3, and rewrites the index.

poetry run export-models-to-yaml # Python models → canonical schema.yaml
poetry run upload-yaml-to-s3 # validate, publish, reindex

Consumers of the registry pick the changes up on their next run, without any intermediate rebuilds or redeploys. So the rest of my post is mostly about what happens inside steps 3 and 5.

Schema identity: the domain/subdomain/entity pattern

I was very opinionated and rigid about the structure of the registry, and not just about how to define the contracts themselves. Every schema gets a unique dotted name composed of domain, zero or more subdomains, and an entity name:

 * lab.analytics.cell_count
 * lab.upstream.bioreactor_run
 * operations.equipment.instrument
 * enterprise.finance.purchase_order

This naming scheme is enforced structurally. Schemas live at file paths that mirror their names:

registry/contracts/domains/lab/analytics/cell_count/v1_0/schema.py

The @register_dataframe_schema decorator captures the identity at class definition time:

@register_dataframe_schema(
 domain="lab",
 subdomains=["analytics"],
 entity="cell_count",
 version="1.0",
 description="LIMS cell count assay table.",
 meta=SchemaMeta(
 tier="bronze",
 alignment="source",
 source_system="lims"
 )
)
class LIMSCellCountSchema(ContractMixin, LIMSBaseSchema):
 run_row_id: str = pa.Field(coerce=True)
 sample_id: str = pa.Field()
 vcd: Optional[float] = pa.Field(nullable=True, coerce=True)
 viability: Optional[float] = pa.Field(nullable=True, coerce=True)
 run_date: Timestamp = pa.Field(nullable=True, coerce=True)

The decorator is thin and only builds a frozen SchemaInfo, derives the dotted name from its parts, and stamps both onto the class:

def _register(cls):
 info = SchemaInfo(
 domain=domain,
 subdomains=subdomains,
 entity=entity,
 version=version,
 owner=owner,
 description=description,
 )
 cls.__schema_info__ = info
 cls.__schema_meta__ = meta or SchemaMeta()
 return cls

SchemaInfo.__post_init__ does a quick n’ dirty validation by rejecting any domain, subdomain, or entity containing ., /, or -, since those would corrupt the dotted name or the path it maps to, and then joins the parts into name. The more expensive checks, that the folder version matches the declared version (v1_0 → 1.0) and that the directory path matches the dotted name, run at export time, not at import time:

if RegistryConfig.enforce_domain_name:
 confirm_schema_name_matches_module(cls=cls, name=info.name)

if RegistryConfig.enforce_path_version:
 confirm_schema_version_matches_module_version(cls=cls, version=info.version)

Enforcing contract path structure at import time makes the contract classes impossible to define anywhere else, which is a problem the first time a scientist wants to draft a schema in a notebook to see what it looks like. Doing it at export means the rule is enforced on everything that gets published, while drafting stays cheap. There’s a context manager for the notebook usecase too:

with relaxed_registry_checks():
 @register_dataframe_schema(...)
 class DraftSchema(ContractMixin, BaseDataFrameSchema):
 ...

But there is a tradeoff, such a badly-placed schema fails later than it could. In practice “later” is the export step, which runs in CI on every PR, so nothing badly-placed reaches S3 regardless. Contracts inherit from a base model that sets the validation posture in one place:

class BaseDataFrameSchema(pa.DataFrameModel):
 class Config:
 strict = 'filter'
 coerce = True

strict='filter' drops columns not in the schema rather than raising, which for source-system data is almost always what you want. The LIMS base adds drop_invalid_rows = True and the row identity columns every LIMS table carries.

Versioning: semantic, path-encoded, and strict

Schema versions follow semantic versioning (1.0, 2.1, 1.0.1) and are encoded into the directory structure as Python-safe package names:

v1_0 → 1.0
v2_1 → 2.1
v1_0_1 → 1.0.1

The conversion is bidirectional and validated in both directions. When I need to add a new version of a schema, I create a new directory alongside the old one. Both continue to exist and be served by the registry.

cell_count/
 v1_0/
 schema.py
 schema.yaml ← auto-generated
 v2_0/ ← new version; v1_0 still works
 schema.py
 schema.yaml

Version resolution supports three modes: a specific version ("1.0"), "latest" (resolved by semantic comparison at read time), and None (all versions). This last mode is useful when you need to understand the full version history of a schema. Resolving latest uses packaging.version.Version for comparison rather than string sorting, so 10.0 sorts above 9.0.

for strver in vers:
 semver = Version(strver)
 # don't allow pre-releases, or dev-releases
 if semver.pre or semver.dev:
 continue
 parsed.append((semver, strver))

So I can publish 2.0.0rc1, point a pipeline at it explicitly to test, and know that nothing pinned to latest will pick it up by accident.

Export pipeline: python → YAML

Rather than maintaining YAML files by hand, the build step automatically discovers and converts decorated Pandera models. Discovery works by walking the file tree looking for schema.py files, dynamically importing them as namespace packages (which avoid collisions between modules at the same relative path), and inspecting each imported module for classes that subclass both pa.DataFrameModel and ContractMixin and have a populated __schema_info__. The code builds a unique dotted module name from the full path:

def _path_to_module_name(filepath: Path, package_root: Path) -> str:
 rel = filepath.resolve().relative_to(package_root.resolve())
 parts = rel.with_suffix("").parts
 return '.'.join(parts)
 # → "contracts.domains.lab.analytics.cell_count.v1_0.schema"

Each import gets a unique name in sys.modules to prevent collisions (ideally we wouldn’t be naming schemas the same name, but this approach allows for it). Setting mod.__package__ to the parent of that dotted name keeps relative imports inside the schema file working.

Pandera fields are serialized into a YAML-friendly dictionary, and checks that can be represented cleanly (e.g. comparisons and membership tests) are serialized. Anything else is skipped with a warning, since they’re harder to represent. Not ideal if you do want some more complicated validation logic, but works for now – though I think Great Expectations is probably better for these sorts of checks than Pandera is.

# pa.Check.greater_than(0) → {"gt": 0}
# pa.Check.isin(["A", "B"]) → {"isin": ["A", "B"]}
# lambda df: df["x"] > df["y"] → warning, skipped

The serializer returns a list per check rather than a single dict, because some Pandera checks are compound. in_range has to expand into a ge entry and an le entry, since the YAML vocabulary has no range primitive. The resulting YAML is human-readable and versionable:

schema:
 name: lab.analytics.cell_count
 domain: lab
 entity: cell_count
 version: "1.0"
 owner: data-platform
 schema_type: dataframe
 description: LIMS cell count assay table.

meta:
 tier: bronze
 alignment: source
 source_system: LIMS

columns:
 - name: run_row_id
 dtype: str
 nullable: false
 coerce: true
 - name: viability
 dtype: float64
 nullable: true
 coerce: true
 checks:
 - ge: 0.0
 - le: 1.0

Canonical dtype vocabulary

My initial version of a Python-to-YAML-to-Python round trip wrote out whatever dtype spelling the contract author (a.k.a. me) used. Pandera and pandas and typing will happily provide float, float64, double, np.float64, typing.List[float], and Timestamp for what are actually three types. I didn’t want consuming services to have to deal with this ambiguity. The registry now defines a closed dtype vocabulary:

scalars: str, int32, int64, float32, float64, bool, datetime64[ns]
lists: List[<scalar>] (element one of int/float/str/bool)

Author-side spellings are normalized to that vocabulary in exactly one function, canonicalize_dtype, called at export time:

_SCALAR_ALIASES: Dict[str, str] = {
 "int": "int64", "integer": "int64",
 "float": "float64", "double": "float64",
 "str": "str", "string": "str",
 "bool": "bool", "boolean": "bool",
 "datetime": "datetime64[ns]",
 "timestamp": "datetime64[ns]",
 "date": "datetime64[ns]",
 # ... plus the canonical spellings mapping to themselves
}

Publishing contracts

The dtype vocabulary is only a contract if something enforces it, so publishing to S3 runs every document through a hard gate first:

def assert_canonical_dtypes(doc, source="<doc>") -> None:
 schema = (doc or {}).get("schema", {})
 if schema.get("schema_type") != "dataframe":
 return

 problems = []
 for col in (doc.get("columns") or []):
 dtype = col.get("dtype")
 if dtype is None:
 continue
 try:
 canonical = canonicalize_dtype(dtype)
 except UnknownDtypeError as e:
 problems.append(f"column {col.get('name')!r}: {e}")
 continue
 if not is_canonical(dtype):
 problems.append(
 f"column {col.get('name')!r}: dtype {dtype!r} is not canonical "
 f"(expected {canonical!r}), re-export this schema."
 )

 if problems:
 raise ValueError(f"Refusing to publish {schema.get('name')!r}:\n - " + "\n - ".join(problems))

The two rejection cases are doing different jobs. Rejecting an unknown dtype catches a type consumers can’t deserialize. Rejecting a dtype that is merely non-canonical (one that canonicalize_dtype could fix) catches something else entirely: it means this YAML was not produced by the current exporter. Someone hand-edited it, or it was generated before the normalization step existed. Silently canonicalizing it at publish time would hide that. Refusing forces a re-export, which keeps the YAML in S3 in sync with the Python that claims to define it. The check acts as sort of a staleness detector.

Schema templates: validating the schemas themselves

Separately from dtypes, each schema document is validated against a JSON Schema template selected by its schema_type:

contracts/templates/
 dataframe/v1_0/schema.yaml
 event/v1_0/schema.yaml
 api/v1_0/schema.yaml

At publish time the templates are themselves checked as valid JSON Schemas with Draft202012Validator.check_schema, compiled once, and then each contract is validated against the template matching its declared type. A schema with no schema_type, or one naming a type with no template, is a hard failure, and wont upload.

This is the layer that lets the registry hold more than DataFrames. dataframe contracts carry columns. api contracts carry request, response, and callback sections. event contracts carry their own shape. They all live in the same tree, share the same identity and versioning rules, and are all published through the same gate, but each is structurally validated against its own template.

Schema metadata: queryable semantic tags

Every schema can carry structured metadata in a SchemaMeta dataclass:

@dataclass(frozen=True)
class SchemaMeta:
 tier: Optional[Literal["bronze", "silver", "gold"]] = None
 alignment: Optional[Literal["source", "canonical", "denormalized"]] = None
 contract_type: Optional[Literal["ingress", "internal", "egress"]] = None
 source_system: Optional[Union[str, List[str]]] = None
 stage: Optional[str] = None

These fields encode the data’s position in a medallion-style architecture. Bronze/silver/gold indicate the refinement level. Alignment captures whether data is still in the shape of its source system, has been canonicalized, or has been denormalized for consumption. Contract type captures directionality in the platform: ingress for uploads and instrument files, internal for the middle of pipelines, egress for published dashboards and APIs. This metadata is queryable:

registry = Registry(s3_bucket="my-data-lakehouse")

# Find all bronze schemas sourced from the LIMS
lims_bronze = registry.find_by_meta(tier="bronze", source_system="lims")

# Find all schemas used as egress contracts
egress_schemas = registry.find_by_meta(contract_type="egress")

# Get all tiers currently in use
registry.get_tiers() # → ["bronze", "gold", "silver"]

This became invaluable for impact analysis. When a source system changed its output, I could immediately identify which schemas were alignment="source" and source_system="lims" and therefore potentially affected.

The same identity travels with the data, not just the registry. ContractMixin can flatten a contract into a string key/value header for stamping onto parquet metadata:

{
 "contract.domain": "lab",
 "contract.entity": "cell_count",
 "contract.version": "1.0",
 ...
}

So a file on disk can answer which contract and which version produced it, without anyone having to consult a pipeline log.

Runtime registry

The Registry class wraps an S3 backend and provides the consumer-facing API. It’s thin by design. Most logic lives in the export pipeline and the contract definitions themselves.

from schema_registry_client import REGISTRY

# Load a schema as a live Pandera model
Model = REGISTRY.get_pandera_model("lab.analytics.cell_count", version="latest")

# Validate a DataFrame against it
validated = Model.validate(df)

# Or get the raw YAML doc
doc = REGISTRY.get_schema("lab.analytics.cell_count", version="1.0")

get_pandera_model builds a class dynamically: each YAML column becomes a Pandera Field plus a type annotation, the check dicts become Field kwargs, and the schema’s config overrides are applied on top of the base model’s config. Because the dtype strings are guaranteed canonical, the annotation lookup is a single flat dict with seven entries and no fallbacks.

The backend reads an index CSV written at upload time rather than doing live list_objects calls:

contracts/index/schema_index.csv # name, key, version, date, schema_type
integrations/index/spec_index.csv

Listing names, resolving versions, and enumerating domains are all filters over that dataframe. It loads once and stays cached on the backend instance. This keeps latency predictable and avoids a per-request S3 API call every time something enumerates the registry. The cost is that the index is a build artifact like everything else: it’s rewritten on every publish, so it’s accurate as of the last upload and no fresher.

Alongside the contracts, the registry stores integration specs that map a contract to its physical source table: schema and table name, a field_to_source column mapping, and the created/modified columns used for date partitioning. That’s what lets a pipeline say “load this contract for this date” without knowing anything about the source system’s table layout. It’s also the seam where a source system renaming a column becomes a spec change rather than a pipeline change.

API

The registry is also served over HTTP by a FastAPI app deployed as an AWS Lambda function, with Cognito authentication applied as a global dependency and a small set of public paths for health and docs.

Method	Path	Description
`GET`	`/domains/`	List all domains
`GET`	`/schemas/`	List schema names (`?domain=`, `?version=`)
`GET`	`/schemas/{name}`	A schema document (`?version=latest`)
`GET`	`/schemas/{name}/versions`	List versions for a schema
`GET`	`/schemas/{name}/diff`	Diff two versions (`?from=&to=`)
`POST`	`/schemas/{name}/validate`	Validate a payload against a schema section

The diff endpoint answers “what actually changed between 1.0 and 2.0”, which is helpful if something starts breaking after publishing schemas. The validate endpoint lets a service check a payload against an api-type contract over HTTP, without installing the client or writing Python at all. It refuses if you point it at a dataframe schema, since there’s no meaningful section to validate against.

Mostly, though, the API exists so that the registry is browsable by people who aren’t going to pip install anything. Being able to send a scientist a URL that shows exactly what a table contains has done more for adoption than any amount of documentation. I ended up building a nice dashboard that displayed datatable-specific contracts as tables, and let’s users browse by domain, subdomain(s), entity, and version. It’s much easier to look at than a YAML file or Pandera class.

Closing toughts

A schema registry is about making implicit contracts explicit. Before building this, the “contract” between a pipeline and its consumer (e.g. dashboards, APIs) was whatever the pipeline happened to output on any given run. Afterwards, it was a versioned, machine-readable document that both sides could validate against independently. None of this is domain-specific, and these same patterns would work for any environment where you need versioned, queryable, code-first schema definitions that can be serialized and distributed. I’ve incorporated contracts for everything from scientific instruments, to finance, to ERP domains in our registry.

I underestimated how much work the enforcement would be, as opposed to the contract definitions themselves. Defining schemas as Python classes is easy, and scientists who are somewhat Python-saavy can do this easily as well. Making it impossible to publish a schema that consumers can’t read, or to end up with two spellings of the same type, or to quietly delete something a pipeline still depends on, took a lot longer, and is what makes the system robust. For our small team in a domain with complex, heterogeneous data, implementing this registry and schema enforcement has been a huge boon.

I work on data platform infrastructure at a biologics company. The patterns in this post are generalized from production code, with domain-specific details abstracted away. The core framework concepts are shareable.

CI/CD Part 2: Docker Images and Container Registries

Tue, 30 May 2023 02:14:14 -0700

This is the second of two posts on designing Gitlab CI/CD pipelines. The first covered the anatomy of a .gitlab-ci.yml file, conditional jobs, and setting up the .pypirc and .netrc files that let a pipeline build a Python package and push it to a registry.

This post covers what happens after that: writing a Dockerfile to build an image from that package, and pushing the image to a container registry, both the Gitlab one and AWS ECR.

Multi-stage Docker builds

I’m using what’s called a “multi-stage” build. This is exactly what it sounds like – it breaks the process of building a Docker image into multiple stages. In doing so, we often have the benefit of a final image that is smaller than a single-stage build, because we only include the artifacts needed to run our containerized application.

Similarly, we can leverage multi-stage Docker builds to minimize duplicated code in Dockerfiles. For example, let’s say we have a scenario where we want to build an image for a Production environment as well as a Test environment. The Test environment might include some additional dependencies, scripts, exports, etc. that the Production environment doesn’t. Instead of creating two Dockerfiles, one for each environment, we can define a single stage that encompasses the overlapping parts of both the Production and Test images, and then define the extra stuff in a separate stage to build the Test image.

In the example below, we have a three-stage Docker build, with stage names:

base: sets up some basic environment variables
python-deps: installs your package and creates a virtual environment
runtime: the actual application you want to run, with only the necessary files for running it

Create a base image

# set base image
# bigger base images yield slower image load times, and have more security vulnerabilities
FROM python:3.9-slim as base

# install virtual environment in ${project_dir}/.venv
ENV PIPENV_VENV_IN_PROJECT 1
# dont write .pyc files
ENV PYTHONDONTWRITEBYTECODE 1
# get some more information about faults when building images
ENV PYTHONFAULTHANDLER 1

Install package dependencies

If you were building your Docker image locally, you’d have access to any authentication tokens or SSH keys necessary to pull from remote or private repositories. However, Docker is naive to these variables – we have to explicitly provide them at build time. Within the Dockerfile, we define three environment variables using the ARG keyword:

CI_DEPLOY_USER
CI_DEPLOY_PASSWORD
CI_JOB_TOKEN

If you think that these variables look familiar, you’re right. They’re the same pre-defined variables that exist in the context of a Gitlab CI/CD pipeline that act as authentication tokens for a .pypirc file and .netrc – they’re utilized by the setup_tokens.sh script to set up the .pypirc and .netrc files within the Docker image.

# image for installing dependencies
# we only need .venv and app `runtime` image, not all other bloat
FROM base AS python-deps

####################################
# ----------------------------------
ARG CI_DEPLOY_USER
ARG CI_DEPLOY_PASSWORD
ARG CI_JOB_TOKEN
# ----------------------------------
####################################

These variables are available outside of the Docker image, but not within the image itself, so we need to “show” them to Docker at build time via the --build-arg flag:

docker build --build-arg CI_DEPLOY_USER=$CI_DEPLOY_URDER \
 --build-arg CI_DEPLOY_PASSWORD=$CI_DEPLOY_PASSWORD \
 --build-arg CI_JOB_TOKEN=$CI_JOB_TOKEN \
 ...
 ...

Now they are contained within the Docker image and can be provided to the setup_tokens.sh script, which then allows us to pull packages down from our remote package registry. We also no longer need the SSH keys, since we’re authenticating through Gitlab itself.

# install pipenv in `python-deps` image
RUN python3 -m pip install pipenv
RUN apt-get update \
 && apt-get install --yes --no-install-recommends gcc g++ libffi-dev

# Dependency installation looks a little different for local packages
WORKDIR /home/app


# copy files to `python-deps` image
COPY setup.py setup_tokens.sh Pipfile Pipfile.lock ./
# copy over application-specific code that you want to install
# this is unique to my specific project -- use your own directories here
COPY templateci/ templateci/

# run setup_tokens script to setup .pypirc and .netrc within image
RUN chmod +x ./setup_tokens.sh && ./setup_tokens.sh

# authentication tokens are now available to pipenv
RUN python3 -m pipenv install --deploy --dev

# get rid of unnecessary libraries after install
RUN apt-get autoremove --yes gcc g++ libffi-dev \
 && rm -rf /var/lib/apt/lists/*

Create your final runtime image

Above, we created the virtual environment that allows our application to run. As such, we no longer need the raw source code or any other random files that were contained in the original project directory that might have been needed to build the virtual environment. Now we create a stage called runtime in which we copy over the generated virtual environment from the previous stage

# image for running the application
FROM base AS runtime

# Copy virtual environment from `python-deps` image to `runtime` image
COPY --from=python-deps /home/app/.venv /.venv
# add virtual environment to PATH
ENV PATH="/.venv/bin:$PATH"

# Create new user -- app will run as new user
RUN useradd --create-home -u 1099 user
WORKDIR /home/user/app
USER user

COPY . .

CMD ["python3", "-m", "pytest"]

This is the actual image that gets run when we call

docker run ${IMAGE_NAME}

Pushing to AWS ECR

The conditional job at the end of the first post pushed an image to the Gitlab Container Registry. Pushing to a remote AWS Elastic Container Registry (ECR) instead follows the same structure, with a different base image and a different authentication step.

Setting up AWS variables

To build images, tag them, and push them to the remote AWS ECR, I used the definition of a CI/CD job below. In addition to pre-defined variables that are set internally by Gitlab, we can also manually pre-define variables. In this case, I’ve set a few that allow me to interact with AWS via the command line:

AWS_DEFAULT_REGION: self-explanatory
ECR_REPO_LAMBDA: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${YOUR_ECR_REPO_NAME}
AWS_ACCOUNT_ID: AWS account ID
AWS_ACCESS_KEY: this is the information contained in the downloaded *.pem file
AWS_SECRET_ACCESS_KEY: this is the information contained in the downloaded *.pem file

To set variables that are accessible by CI/CD jobs, go to your Project/Group > Settings > CI/CD > Variables > Expand and define the variables of interest:

If you define these variables at the Gitlab Group level, they will be propagated down to the project level, so long as the Project falls under the Group scope.

Pushing to AWS ECR via CI/CD job

Below, we define the actual CI/CD job. There were two aspects here that I needed to solve. First, I needed access to a Docker-in-Docker build image e.g. an image that had Docker installed. And second, this image also needed to have the AWS CLI tool installed. To that end, I used the bentolor/docker-dind-awscli image.

build-image-ecr:
 stage: deploy 
 image: bentolor/docker-dind-awscli
 services:
 - docker:dind
 variables:
 # convenience variable indicating name of the image with respect to the ECR repo and unique tag ID
 IMAGE_TAG: $ECR_REPO_LAMBDA:$CI_COMMIT_SHORT_SHA

 before_script:
 - docker info
 # authenticate docker with your AWS ECR account
 - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
 # will push Docker image to AWS ECR
 script:
 # build the docker imagae
 - docker build --compress -t ${IMAGE_TAG} .
 # tag the image with a unique name
 - docker tag ${IMAGE_TAG} $ECR_REPO_LAMBDA:latest
 # push the image to the ECR
 - docker push ${IMAGE_TAG}

 # here, we only build and push the image if this is a merge event into the "main" branch
 rules:
 - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"

And voila! You have now pushed your built image to a remote container registry!

CI/CD Part 1: Gitlab Pipelines and Package Registries

Tue, 02 May 2023 02:14:14 -0700

I recently developed a template workflow to help our team adopt a CI/CD-based development strategy. Many of our web applications and tools were based on simple repository structures. With growing datasets and ever-increasing use by outside teams, we found ourselves needing to add new features more frequently to many of these tools and believed that continuous integration and deployment could help us not just develop more quickly, but also more intelligently. Since we use Gitlab to store our code, we decided to use the Gitlab CI/CD tools.

Documentation on much of this process was scattered and/or sparse, so I decided to put what I learned and implemented into a more coherent set of notes. This post covers the anatomy of a .gitlab-ci.yml file, how jobs and stages fit together, how to make a job conditional, and how to authenticate against a package registry so a pipeline can build a Python package and push it there.

The second post covers the other half: writing a multi-stage Dockerfile to build an image from that package, and pushing the image to a container registry.

Setup

I did all of my testing using my personal Gitlab account. To separate things out, I created a new Project called “Package Registry”, as well as a test repository that was used for building a local Python project called “TemplateCI”.

The “Package Registry” Project serves as just that – an all-inclusive location for any software packages your CI/CD pipelines build. You can find the built packages by clicking ${Project Name} > Deploy > Package Registry. Every “Project” in Gitlab has the ability to store packages in its own registry, but I felt it cleaner to store everything in one repo. Similarly, the actual code that I’ll be packaging will be stored in the “TemplateCI” repo.

Authentication: .pypirc and .netrc

In order to build packages and push them to a remote package registry, we use the build and twine packages. build generates a package, and twine pushes this package to a registry (or “index”). twine requires access to authentication usernames, passwords, and a registry URL in order to do so. twine can access these tokens from a .pypirc file – the tokens are generated by the registry, and ensure that the submitting user has permissions to perform a certain action.

Other processes, such as pulling or pushing code from a remote repository, often require additional usernames and passwords. In order to alleviate the need to consistently provide these variables at request time, we can save them in a .netrc file.

These are straightforward to set up locally. But we also need to set these up to ensure a properly functional CI/CD workflow. I’ve put together a basic script, called setup_tokens.sh that does just that:

setup_tokens.sh

#!/bin/bash

# generate .pypirc file
echo "[distutils]
index-servers =
 personal

[personal]
repository = https://gitlab.com/api/v4/projects/$PACKAGE_REGISTRY_ID/packages/pypi
username = $CI_DEPLOY_USER
password = $CI_DEPLOY_PASSWORD" > ~/.pypirc

# generate .netrc file
echo "machine gitlab.com
login gitlab-ci-token
password $CI_JOB_TOKEN" > ~/.netrc

The .pypirc refers to your Project Registry via a previously generated authentication token and password, and allows your to build and upload Python packages to that registry.

The .netrc file enables you to pull private packages from that same registry. In the context of our work, we’ll want to build and push packages to the registry first so that they are available for pulling. For example, in the Pipfile for this template project, we have the following:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${CI_DEPLOY_USER}:${CI_DEPLOY_PASSWORD}@gitlab.com/api/v4/projects/${$PACKAGE_REGISTRY_ID}/packages/pypi/simple"
verify_ssl = true
name = "personal"

We see the same user authentication happening, along with the reference to the Package Registry ID variable. For local installation of your package, and in order to make sure that your Pipfile and Pipfile.lock are in sync, you’ll need to define the following local environment variables:

CI_DEPLOY_USER: generated user token
CI_DEPLOY_PASSWORD: generated token password
PACKAGE_REGISTRY_ID: the ID of the repository that you created that will store your packages

Basic jobs

The basis of a Gitlab CI/CD pipeline is the .gitlab-ci.yml file, which is composed of a set of explicitly-defined and temporally-ordered “stages”. A stage is composed of a set of “jobs”. Jobs are the workhorses of the CI/CD pipeline, and define explicit tasks that a CI/CD pipeline runs. By default, all jobs from one stage run in parallel, unless specified otherwise (using the needs keyword as an attribute of a job induces a temporal directed acyclic graph – jobs can be made “dependent” on the successful completion of other jobs within a stage). In this example, I’ve defined three stages:

run-unit-tests
build-package
build-image

Any job associated with the run-unit-tests stage will run to completion (or failure) PRIOR TO ANY job in the stages build-package and build-image starting. If all jobs in the run-unit-tests stage complete successfully, then the next stage (build-package) will begin. We define individual jobs, and give them a stage attribute. Stages define the rough ordering of jobs. Each job runs in the context of an “image” or environment. We can set a global image and define stages as

# global CI/CD image
image: python:3.9-slim

# stages of this example pipeline
stages:
 - run-unit-tests
 - build-package
 - build-image

 variables:
 LC_ALL: C.UTF-8
 LANG: C.UTF-8

or define the image as an attribute of a job. Gitlab CI/CD by default uses Docker images in which to run jobs. Setting the image is analogous to using the FROM command in a Dockerfile. We’ve also set some global variables, here the LC_ALL and LANG variables.

To “run” Gitlab pipelines for the purpose of CI/CD, we use “runners”, which are build instances installed on a server. Gitlab offers “shared” runners (use of these is free if you use www.gitlab.com, but you need to register a credit card to prevent abuse of Gitlab resources). You can also register your own device(s) to act as a Gitlab runner.

Below are examples of two jobs in the run-unit-tests stage. These two jobs are effectively the same code, apart from the unique unit tests that they run. However, we’ve made the job unit-tests-2 dependent on the output of the job unit-tests-1 (see the needs keyword of unit-tests-2). Both jobs use the global python:3.9-slim image. We can run some “setup” stuff (before_script), run an actual script (script), and run clean up (after_script, not shown) – these delineations (before, during, after) are for organizational purposes, and not due to any explicit functional differences in the delineations.

# setup_tokens.sh is the script from the previous section.


# example job #1
unit-tests-1:
 stage: run-unit-tests
 before_script:
 - chmod +x ./setup_tokens.sh; ./setup_tokens.sh
 - python3 -m pip install pipenv
 - apt-get update
 - apt-get install --yes --no-install-recommends gcc g++ libffi-dev
 - python3 -m pipenv install --deploy --dev
 # run first set of unit tests
 script:
 - python3 -m pipenv run pytest -k 'test_examples1.py'


# example job #2
unit-tests-2:
 stage: run-unit-tests
 before_script:
 - chmod +x ./setup_tokens.sh; ./setup_tokens.sh
 - python3 -m pip install pipenv
 - apt-get update
 - apt-get install --yes --no-install-recommends gcc g++ libffi-dev
 - python3 -m pipenv install --deploy --dev
 # run second set of unit tests
 script:
 - python3 -m pipenv run pytest -k 'test_examples2.py'
 # wait for job 1 to finish
 needs: [unit-tests-1]

Conditional pipeline jobs

The above jobs are relatively simple and will run every time you push a repository to Gitlab. However, sometimes, we might only want to run a job if certain conditions are met. For example, we might only want to build a package from the main branch, or only after a merge request is made. To this end, we can add “rules” to a job that restrict when it is actually run.

Below is a more complicated job example. The overarching goal of this job is to build a Docker image from a local Python project and push the image to the Gitlab Container Registry. There’s a lot going on here, so I’ll break it up into pieces, but here is the whole job:

# Conditional job
# Building a docker image and pushing this container to a container registry
# Link to main image: https://github.com/bentolor/docker-dind-awscli

# Conditions:
# --- merge request events
# --- target branch of merge request is "main"

# job name
build-image-glcr:

 # stage of pipeline
 stage: build-image
 # image that job is based on
 image: docker:20.10.16
 # sub-services of job
 services:
 - docker:20.10.16-dind
 # variables available to the job
 variables:
 DOCKER_TLS_CERTDIR: "/certs"
 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

 # some setup scripts -- here, just making sure docker is available
 before_script:
 - docker info

 # meat of the job -- authentication, building, run, push
 script:
 - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
 - docker build 
 --build-arg CI_DEPLOY_USER=$CI_DEPLOY_USER 
 --build-arg CI_DEPLOY_PASSWORD=$CI_DEPLOY_PASSWORD 
 --build-arg CI_JOB_TOKEN=$CI_JOB_TOKEN
 -t $IMAGE_TAG .
 - docker run $IMAGE_TAG
 - docker push $IMAGE_TAG

 # job conditions
 rules:
 - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"

This job is associated with a new stage called build-image that will run as the last stage of this example CI/CD pipeline. Without going into the specifics of the Dockerfile just yet, this stage builds and pushes a Docker image to a remote repository. We defined the job image as docker:20.10.16 and an additional “service” attribute as docker:20.10.16-dind where “dind” means “Docker-in-Docker”. The Docker-in-Docker feature allows an image to run Docker itself (pretty meta, huh). The idea here is to instantiate a job from a specific Docker container (“image”) which itself has Docker installed (“dind”), which will allow the docker:20.10.16 image to build another Docker container. We’ve also defined some variables:

DOCKER_TLS_CERTDIR: needed to allow the larger scale image to communicate with a service (honestly, I don’t quite understand this and documentation on CI/CD “services” is sparse)
IMAGE_TAG: refers to the container destination and the image “name” – this just makes our lives easier by turning into a variable what would otherwise be a really long string

#### Build Docker image and push to Gitlab Container Registry
build-image-glcr:
 stage: build-image
 image: docker:20.10.16
 services:
 - docker:20.10.16-dind
 variables:
 DOCKER_TLS_CERTDIR: "/certs"
 IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG

Next up, we have all the script stuff. You’ll probably recognize most of the docker ${command} commands. We first “authenticate” the current job with the Gitlab container registry (apparently there are a variety of ways to “authenticate” with Docker using one set of variables or another – the way below is the way I was able to get working, but there are other solutions). We then build the Docker image, run the image, and push the image to the container registry.

script:
 - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
 - docker build 
 --build-arg CI_DEPLOY_USER=$CI_DEPLOY_USER 
 --build-arg CI_DEPLOY_PASSWORD=$CI_DEPLOY_PASSWORD 
 --build-arg CI_JOB_TOKEN=$CI_JOB_TOKEN
 -t $IMAGE_TAG .
 - docker run $IMAGE_TAG
 - docker push $IMAGE_TAG

You’ll notice a bunch of variables that I didn’t explicitly define anywhere:

CI_REGISTRY_USER: username for project (I think we can also use CI_DEPLOY_USER, though it looks like there might be some things to figure out with branch / variable protections / non-protections)
CI_REGISTRY_PASSWORD: defaults to CI_JOB_TOKEN value (this value is ephemeral e.g. valid only for one job at a time I think? I think we can also use the CI_DEPLOY_PASSWORD for a longer-lived alternative, though it looks like there might be some things to figure out with branch / variable protections / non-protections)
CI_REGISTRY: defaults to https://gitlab.com/${group}/${project-name}/container_registry
CI_DEPLOY_USER: generated user token
CI_DEPLOY_PASSWORD: generated token password
CI_JOB_TOKEN: see documentation here

These are all “predefined” variables, meaning they already exist in the Gitlab CI/CD context as part of having a www.gitlab.com account, without you explicitly defining them. However, I did run into some issues using CI_DEPLOY_USER and CI_DEPLOY_PASSWORD. In addition to having predefined variables provided by Gitlab CI/CD, we can also manually predefine variables for a whole Gitlab Project or for a whole Group. Go to the page for your Project/Group > Settings > CI/CD > Variables > Expand. For example, I defined the CI_DEPLOY_USER and CI_DEPLOY_PASSWORD variables for my Group. These variables are the Group-level authentication tokens and are now made accessible to all Gitlab CI/CD jobs running under this Group. Additionally, although not shown here (because it’s used within the setup_tokens.sh script), we’ve also defined an environment variable called PACKAGE_REGISTRY_ID that tells the CI/CD pipeline where to build and push packages.

And finally, what we’ve been waiting for, conditional pipeline jobs:

rules:
 - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "main"

As part of this job, I’ve defined a rule that indicates that this job should only be run 1) upon a merge request event (i.e. if you click “Create Merge Request” in the console), and 2) if the name of the target branch of that merge request event is main. For example, if I create a new branch called dev off of main and create a merge request in the Gitlab console, this job will run. However, it’s important to note that, if you push a commit to the dev branch after creating the merge request, this job will still run, even if you haven’t created a new merge request e.g. CI_PIPELINE_SOURCE == “merge_request_event” will always default to TRUE after the first merge request event, as long as the source branch is still actively being developed. Additionally, the job itself for this will run the source branch code, not the target branch code. For Gitlab Premium users, there is an additional criterion called a “merged_result_event”, which would run the target branch (main) code after merging the source branch (dev) into the target branch.

Building and pushing a package

Building a package locally is straightforward, but doing so within a Gitlab CI/CD pipeline is a little more complicated. But, we can imagine adding this type of task to a CI/CD pipeline, and conditioning it on a merge request (or something of that kind).

I’ve called this job build-package and it belongs to a stage also called build-package. We first set up the .pypirc and .netrc files in image running the job, and then install the build and twine libraries in the before_script attribute. Then, using the script attribute, we build our package, and push it to our package registry (we’ve defined the registry in our .pypirc file – here, it’s referred to as the “personal” registry.)

image: python:3.9-slim
variables:
 PACKAGE_REGISTRY_NAME: "personal"

stages:
 - build-package

#### BUILDING PACKAGE AND PUSHING TO GITLAB PACKAGE REGISTRY
build-package:
 stage: build-package
 before_script:
 - chmod +x ./setup_tokens.sh; ./setup_tokens.sh
 - apt-get update
 - apt-get install --yes --no-install-recommends gcc g++ libffi-dev
 - python3 -m pip install build twine
 script:
 - python3 -m build
 - python3 -m twine upload --repository ${PACKAGE_REGISTRY_NAME} dist/* --verbose

Some of the (many) external links I used in this process:

Predefined CI/CD variables
Adding variables to the Gitlab CI/CD context
Setting up a ~/.netrc file with Gitlab credentials
Executing a bash scripting within a Gitlab CI/CD pipeline
Passing variables to Docker image at build time
Building Docker image and pushing to registry
Authenticating container registries
Setting up a ~/.pypirc files with Gitlab credentials

Lab Meeting: pip and the Python Packaging Index

Sun, 07 Jun 2020 10:20:09 -0700

What follows are the contents of part of a lab meeting presentation I gave recently. The topic of the meeting was “Python for Neuroimaging”, where I covered basic software development tools that brain imaging scientists might be interested in.

Creating Python Packages

In this lesson, I’ll show you how to build your own Python package that you can then install locally or upload to the Python Packaging Index (for those of you familiar with R, think CRAN, but for Python).

I’m going to be basing a lot of the material off of this documentation, but will also show a real example using some of my own personal code.

What are packages?

I’m sure most of you are familiar with packages and libraries already, either from Matlab, R, or Python. Packages are basically bundles of various snippets of code, i.e. methods, classes, scripts, tests etc. that are bundled together to perform some function. Generally (hopefully), there is coherence to what these snippets of code do – they should interact together in some way or relate to some overarching computational goal.

Within a package, you can have different groupings of code, where each grouping does some unique or discrete computing. These groupings are called submodules. A common submodule in many packages is an Input / Output (io) module that will read and write data that this package interacts with or produces. Another common submodule is often related to plotting the outputs of your code. And then almost always, there are submodules that perform the brunt of the algorithmic work. So inside modules, you’ll find snippets of code that relate to the goal or concept of the module.

Think of a package as a toolbox with a bunch of drawers, each with a label: wood-working, welding, gardening, flooring, etc. These drawers are submodules. You can tell by their names that they each cover certain topics. Each drawer contains a set of tools: wood-working might contain saw, nail, sandpaper, wood glue, while welding might contain solder, flux, oxygen, glove. These tools are the functions, classes, and scripts that relate to that submodule.

Overall, this toolbox performs some stuff related to construction, homebuilding, repair, and has discrete bundles of code useful for a variety of those tasks.

Directory structure for a Python package

Here we examine the skeleton of a package. All packages follow this basic structure.

pkg_name
|-- __init__.py
|-- LICENSE
|-- pkg_name/
| |-- submodule_a/
| | |-- __init__.py
| | |-- a_1.py
| |-- submodule_b/
| | |-- __init__.py
| | |-- b_1.py
| | |-- b_2.py
|-- README.md
|-- setup.py
|-- test/
| |-- __init__.py

__init__.py is a required file that allows your package to be imported. The only __init__.py file that needs to contain anything is the highest-level file. The others can be empty, but they must exist. Here are the contents of the highest-level __init__.py file:

__all__ = [
 'a_1', 'b_1', 'b_2'
]

from .submodule_a import (a_1)
from .submodule_b import (b_1, b_2)

LICENSE tells other users / individuals in what capacity they are allowed to use your code.

README.md describes how to use your code, and often contains examples. This is a markdown file, but can generally be any type of markup language.

test/ is a directory in which you would want to write unit tests for your code.

setup.py is what allows you to install your package. It’s a set of instructions that get supplied to setuptools package.

from setuptools import setup, find_packages

with open("README.md", "r") as fh:
 long_description = fh.read()

 setup(
 name='pkg_name',
 version='0.1.0',
 author='Kristian M. Eschenburg',
 author_email='keschenb@uw.edu',
 packages=find_packages(),
 scripts=[],
 url="https://github.com/kristianeschenburg/pkg_name",
 license='LICENSE.txt',
 description='An awesome package that does something',
 long_description=long_description,
 install_requires=[
 "numpy"
 "pytest",
 "matplotlib",
 ],
)

Compiling, installing, and uploading your package

1. Register on PyPi

Once we’ve done all this, we’re just about ready to create our Python package and upload it to pypi.org. But first, we need to create an account. For testing purposes, we’ll create a test account here, but the process is the same.

After you create your account, we need to create an API token, that will allow us to upload files to either Test PyPi or PyPi (depending on what we’re doing) – the following steps are the same, regardless.

Under your Test PyPi account, click your username in the top right, go to Account Settings:

Scroll down and click Add API Token:

Follow the instructions there, making sure to select “Entire Account” option under the Scope tab.

DO NOT CLOSE THIS WINDOW WHEN THIS IS COMPLETE

Next, type

cd $HOME
touch .pypirc

and using your favorite text editor, enter the following:

[testpypi]
 repository: https://test.pypi.org/legacy/
 username = __token__
 password = pypi-***

If we were creating a token for PyPi, we’d type:

[pypi]
 repository: https://pypi.org/
 username = __token__
 password = pypi-***

Close your current terminal, and open a new window to refresh your settings. Now, when we go to upload our package to PyPi, we’ll be able to type the commands without needed to supply a username and password directly.

2. Compile your package

First, we need to make sure that a few Python packages are installed. Namely, we need to install pip

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py
pip install -U pip

Then we can install the following packages:

pip install --upgrade pip setuptools wheel # for installing Python packages
pip install tqdm # progress bar package
pip install --user --upgrade twine # for publishing to PyPi

Then, we can compile our package:

python setup.py bdist_wheel

which creates the directories dist, build, and pkg_name.egg-info. The *.egg-info file is basically some zipped meta-data about your package, but we’re really only interested in the *.whl file in dist – “wheels” are a “distribution” format, newly designed to replace “eggs”. I won’t go into it here, but eggs were sort an ad hoc solution to packaging Python code – wheels were part of PEP427 i.e. is actually an “enhancement” to the Python language, and the formal way of packaging Python code.

3. Installing your code

We can install our code locally with:

pip install dist/pkg_name-0.0.0-py3-none-any.whl

If you want to install an “editable” version of your package, do this:

pip install -e .

This will allow you to change your *.py files and have these changes take effect immediately when importing your package, without needing to rebuild each time – but this method installs from the egg distribution, and generally produces larger build files, since the build needs to keep track of your actual source code.

4. Upload your code

We can upload our code to PyPi now using the following command:

python3 -m twine upload --repository testpypi dist/*

Now, if you click “Your Projects” under your account name on PyPi, you’ll see that you project has been uploaded.

** I should note that, any time you want to upgrade your code and upload it to PyPi again, you need to remove all files from the dist directory, increment the version number in the setup.py file – i.e. 0.0.0 –> 0.0.1 – rebuild your package with

bash python setup.py bdist_wheel

Example with personal package

I don’t generally upload my code to PyPi (probably scared bugs in the code, and people finding them, and then thinking I’m terrible at software development, and going down a long spiral of self-deprecation, but I digress…) but I do upload it all to GitHub. In either case, here is a walk-through of packaging some software called pysurface that I use for processing mesh-based data – I use it for adjacency matrices, performing Laplacian smoothing on surfaces, sampling points from mesh triangle simplices, plotting on surfaces… Just some stuff that I find myself doing a lot.

Here is the directory containing all my code:

You’ll see 5 different modules: graphs, operations, plotting, spectra, and utilities, and you’ll note that each module directory has a __init__.py file.

Here is my setup.py file:

from os import path
from setuptools import setup, find_packages
import sys

here = path.abspath(path.dirname(__file__))

with open(path.join(here, 'README.rst'), encoding='utf-8') as readme_file:
 readme = readme_file.read()

with open(path.join(here, 'requirements.txt')) as requirements_file:
 # Parse requirements.txt, ignoring any commented-out lines.
 requirements = [line for line in requirements_file.read().splitlines()
 if not line.startswith('#')]


setup(
 name='pysurface',
 version="0.0.4",
 description="Python package for quickly processing surface meshes.",
 long_description=readme,
 author="Kristian Eschenburg",
 author_email='keschenb@uw.edu',
 url='https://github.com/kristianeschenburg/pysurface',
 packages=find_packages(),
 entry_points={
 'console_scripts': [
 # 'some.module:some_function',
 ],
 },
 include_package_data=True,
 package_data={
 'pysurface': [
 # When adding files here, remember to update MANIFEST.in as well,
 # or else they will not be included in the distribution on PyPI!
 # 'path/to/data_file',
 ]
 },
 install_requires=requirements,
 license="BSD (3-clause)",
 classifiers=[
 'Development Status :: 2 - Pre-Alpha',
 'Natural Language :: English',
 'Programming Language :: Python :: 3',
 ],
)

You can see that I’ve run

python setup.py bdist_wheel

based off the dist, build, and pysurface.egg-info directories. dist contains a file called pysurface-0.0.4-py3-none.any.whl, which is the actual distribution that can be used for installation. I’ve uploaded the code to Test PyPi, and this is what we see:

We can then install the package and all of it’s dependencies from TestPypi via

pip install --index-url https://test.pypi.org/simple/ pysurface

Now I can do something like the following in a Python script:

import pysurface
# or
from pysurface import graphs, spectra
# or
from pysurface.spectra import eigenspectrum

Submitting Batch Jobs with qsub

Tue, 05 May 2020 14:24:17 -0700

I’m fortunate enought to work in a lab with some high-level computing infrastructure. We have a cluster of machines using the Sun Grid Engine (SGE) software system for distributed resource management. The other day, I was searching for how to wrap my Python scripts with qsub so that I could submit a batch of jobs to our cluster. Eventually, I want to be able to submit jobs with dependencies between them, but we’ll start here.

Let’s create an example script that computes the mean of an MRI image. Let’s call the script compute_mean.py:

#!/usr/bin/env python

import argparse
import nibabel as nb
import numpy as np
import pandas as pd


parser = ArgumentParser('Compute the mean of MRI, and save to CSV file.')
parser.add_argument('-i', '--input_image', help='Path to MRI image.',
 required=True, type=str)
parser.add_argument('-o', '--output_csv', help='Output CSV file.',
 required=True, type=str)

args = parser.parse_args()

# read in image
img = nb.load(args.input_image)
# get voxel-wise data
data = img.get_data()

# compute mean
mu = np.mean(data)

# save to csv
df = pd.DataFrame({'mean': [mu]})
df.to_csv(args.output_csv)

The first line of this script, #!/usr/bin/env python tells the script to use the local python environment. In my case, I have a customized installation of Python, along with a bunch packages and libraries that I’ve written and installed that are not available for the rest of my lab (since they’re still in the testing phase or just something I’m experimenting with). This line tells the script to use my Python environment, rather than the default version on our servers.

We can then create a bash wrapper, let’s call in mean_wrapper.sh for a single subject

#!/bin/bash
#$ -M keschenb@uw.edu
#$ -m abe
#$ -r y
#$ -o tmp.out
#$ -e tmp.err

# Compute mean of image
image=$1
output=$2

python compute_mean.py ${image} ${output}

The second and third line here, with the M and m parameters, tell the script to email me once it completes the processing (or if there are any errors). And finally, we can create a wrapper that takes in a list of subjects to process, and the input and output directories, and submits each individual job to the queue using qsub:

#!/bin/bash

subjects=$1
image_dir=$2
output_dir=$3

# we create a variable, as our cluster has 2 different queues to use
# this could be hardcoded though
queue_name=$4

while read subj
do

 image_file=${image_dir}${subj}.nii.gz
 output_file=${output_dir}${subj}.csv

 qsub -q ${queue_name}.q mean_wrapper.sh ${input_image} ${output_file}

done <${subjects}

Here’s an example output from running qstat after submitting a batch of jobs to the cluster:

Example of qstat command, after submitting jobs view qsub.

One thing you’ll notice is the column priority – this is literally a priority queue data structure that I mentioned in my last post on the Watershed by Flooding algorithm. Each job is submitted to the queue with a priority value assigned to it by the SGE software, and the jobs are processed in that order – highest priority first, lowest priority last. Your IT manager can personalize the priority values for specific users or types of jobs, such that they are given preference or moved back in line. This represents an equitable way of distributing compute resources across users in a lab, generally using a first-come, first-serve basis, or restricting users to a certain number of nodes.

Watershed by Flooding: Applied Data Structures

Fri, 01 May 2020 11:12:32 -0700

I’m applying some methods developed in this paper for testing purposes in my own thesis research. Specifically, I have some float-valued data, $F$, that varies along the cortical surface of the brain. Visually, I can see that there are areas where these scalar maps change abruptly. I want to identify these boundaries – eventually, I’ll segment out the regions I’m interested in.

Computing the Gradient Map

The authors use some conventional brain imaging software to compute the gradient of their data. The domain of this data is a triangulated mesh, described by the graph $G = (V, E)$, where $V$ are vertices in Euclidean space and $E$ the edges between these vertices. In short, for a vertex, $v_{i}$, we first need to compute the gradient vector of the scalar field at $v_{i}$. We “unfold” the 3D positions of adjacent vertices onto the tangent plane of $v_{i}$ – which we can do by orthogonally projecting the adjacent vertices onto the affine subspace at $v_{i}$ and then weighting appropriately. We then regress the graph signal (our scalar field) onto these unfolded positions. The $L_{2}$ norm of this vector is the gradient at $v_{i}$.

For each vertex, we have a normal vector to the surface $N$, its spatial 3D coordinates $v_{i} = (x, y, z)$, and a list of its adjacent vertices. We can compute the orthogonal projector onto the affine subspace spanned by $N$, $P_{N}$, and its orthogonal complement, $Q_{N}$, as:

$$\begin{align} P_{N} &= N(N^{T}N)^{-1}N^{T} \\ Q_{N} &= I - P_{N} \end{align}$$

For any vertex, $v_{j}$, we can compute the orthogonal projection onto the affine subspace spanned by $Q_{N}$ as:

$$\begin{align} q(v_{j}) = Q_{N}(v_{j} - v_{i}) + v_{i} \end{align}$$

We generate the vectors

$$ S_{i} = f(i) - \begin{bmatrix} f(v_{1}) \\ f(v_{2}) \\ \vdots \\ f(v_{j}) \end{bmatrix} \in \mathbb{R}^{j} \;\;\; R_{i} = \begin{bmatrix} q(v_{1}) \\ q(v_{2}) \\ \vdots \\ q(v_{j}) \end{bmatrix} \in \mathbb{R}^{j \times 3} $$

where $S_{i}$ is the difference between the scalar value at our vertex $f(v_{i})$ and the vector of adjacent $j$ scalar field values, and $R_{i}$ is the matrix of $j$ orthogonally projected adjacent vertex coordinates. Then we perform least squares regression to solve for $\beta$:

$$\begin{align} S_{i} = R_{i}\beta \end{align}$$

where $\beta \in \mathbb{R}^{3}$, which indicates how much each coordinate axis contributes to variation in the scalar field at $v_{i}$. The gradient value at vertex $v_{i} = \left || \beta \right||_{2}$.

Watershed By Flooding Algorithm

The new scalar field of $L_{2}$ norms is our gradient field, which describes how “quickly” our original data changes at each vertex. We can now apply the Watershed Algorithm to segment our mesh data. In brief, the watershed algorithm treats the gradient field as a topographic map: low-elevation areas (areas with a small gradient) are “water basins”. If we imagine water flooding this map from the bottom up, basins at low elevation will flood first, while areas at higher elevations will fill last. When water basins meet, the water has reached a “boundary” (or ridgeline, if we’re using the topographic map idea).

I’ve implemented an algorithm variant called “Priority Flooding”, using Python’s heapq priority queue data type class. The priority queue is an application of the binary heap data structure – as nodes are added to the heap, the branching process determines where to put nodes (left or right of a current node), based on some value – in the case of the priority queue, this value is the “priority”. We utilize the priority queue because it gives us a principled way to iterate over unlabeled vertices, and, with some auxiliary data structures, is guaranteed to converge. The algorithm proceeds as follows:

Identify local minima, and assign each minimum a unique label
Add directly adjacent vertices of local minima to priority queue (lower gradient -> higher priority)
While the queue is not empty, get the highest-priority item
- If vertices adjacent to this vertex have only one label, assign this vertex to that label
- Else assign this vertex as a boundary vertex
- Add unlabeled adjacent vertices to the queue

import numpy as np
from queue import PriorityQueue

class PriorityFlood(object):

 """
 Class to segment a scalar field using the Priority Flooding
 watershed algorithm.
 """

 def __init__(self):

 self.pq = PriorityQueue()

 def fit(self, gradient, A, M):

 """
 Fitting procedure for watershed algorithm.

 Parameters:
 - - - - -
 gradient: float, array
 map of gradient values of scalar field
 A: dict of lists
 adjacency list of data domain
 M: list
 local minima in scalar field used to seed algorithm
 """

 # initialize empty label vector
 labels = np.zeros((gradient.shape))
 labels[:] = np.nan

 # keep track of items in queue
 in_queue = np.zeros((gradient.shape)).astype(np.bool)

 # assign local minima unique labels
 # add their adjacent vertices to heap
 for i, loc_min in enumerate(M):

 # label local minima
 labels[loc_min] = i+1

 # add neighbors of local minima to p-queue
 for nidx in A[loc_min]:
 in_queue[nidx] = True
 self.pq.put((gradient[nidx], nidx))

 # iterate over p-queue items
 # items assigned to a label or
 # assigned as a boundary
 while not self.pq.empty():

 # get highest priority item
 # mark as not in p-queue
 [gr, idx] = self.pq.get()
 in_queue[idx] = False

 # get item neighbors and their labels
 i_neighbors = np.asarray(A[idx])
 i_nlabels = labels[i_neighbors]

 # get labels of adjacent vertices that are not nan
 nans = np.isnan(i_nlabels)
 unique_labels = np.unique(i_nlabels[~nans])

 # if more than one unique label
 # assign current vertex as border vertex
 if len(unique_labels) > 1:
 continue

 # otherwise assign to water basin
 else:

 # basin assignment
 labels[idx] = unique_labels

 # identify neighbors without labels that arent in the p-queue
 gidx = np.where((nans) & (~in_queue[i_neighbors]))[0]

 # add these to p-queue
 for nidx in i_neighbors[gidx]:
 if not in_queue[nidx]:
 in_queue[nidx] = True
 self.pq.put((gradient[nidx], nidx))

 self.labels_ = labels

Example of PriorityFlooding algorithm, applied to gradient of inferiorparietal region. Shown on inflated and flattened cortical surfaces.

One caveat that came up is that the PriorityQueue class does not check for duplicates – that is, two vertices might share an adjacent vertex, and this vertex might have already been added to the heap. In this case, we would unnecessarily view the same vertices many times. To alleviate this, we create a boolean Numpy array, in_queue, that stores whether an item is already in the queue.

Headfirst into (Uncommented) C++

Mon, 29 Oct 2018 08:12:32 -0700

While most of my day-to-day research entails writing Python code, I also make heavy use of pre-written software. Most software comes pre-compiled, but whenever possible, I like to get access to the source code. I’m going to refer to some modifications I made to pre-existing packages – you can find those in my repository here.

The most-commonly used open-source package for brain imaging is called FMRIB Software Library (FSL), which includes tools for processing MRI data, with applications ranging from motion correction and image registration, to modal decomposition methods, among many others. All of this is made available as a set of pre-compiled C++ binaries.

I needed to modify FSL’s probtrackx2 tool. probtrackx2 is a tool for generating probabilistic tractography. Using diffusion MRI, we can model the movement of water in the brain. At the voxel level, diffusion tends to be high when water moves along neuronal axon bundles, and low when moving against the myelin or in the extracellular matrix – this water movement can be modeled using a variety of approaches.

Diffusion tractography from Biomedical Image Computing Group at USC.

At the simplest level, the diffusion can be modeled as a diffusion tensor, where the eigenvalues of the tensor correspond to the amount of diffusion in the direction of the corresponding eigenvector. At the more complex levels, we can represent the diffusion as a 3D probability distribution function, whose marginal distributions are called orientation distribution functions (ODF), and represent these continuous functions using a spherical harmonics basis set of the ODF. Using probtrackx2, we can sample these ODFs using a Markov Chain Monte Carlo approach and “walk” through the brain. Directions where the diffusion signal is high will be sampled more often, and we can generate a robust representation of the macroscale neuronal structure in the brain using these random walks.

Orientation distribution functions from Vega et al. 2009.

The diffusion signal at the gray matter / white matter interface of the cortex is more isotropic than within the white matter (e.g. the diffusion tensors in these regions are more spherical). To reduce noise in my fiber tracking results due to this low signal, I wanted to be able to force the first steps of the streamline propagation algorithm to follow a specific direction into the white matter, before beginning the MCMC sampling procedure. Essentially what this boils down to is providing probtrackx2 with prespecified spherical coordinates (azimuthal and polar angles) for the first propagation step. More specifically, I computed the initial spherical coordinates using surfaces computed from the mesh curvature flow results of St.-Onge et al. Importantly, I wanted to make use of the probtrackx2 infrastructure as much as possible e.g. I didn’t want to write my own classes for loading in surface data, and wanted to minimally update the members of any other classes I found useful.

Surface-flow seeded tractography from St-Onge et al. 2018.

Jumping under the hood into the probtrackx2 code was a feat. While the software is sophisticated, it is quite poorly documented. As is common with academic code, development generally begins as a way to solve a specific problem in the lab, rather than as a package to be made available for public use. FSL has been around for a while, and grows in complexity all the time, so the initial academic-oriented mindset has somewhat propagated through their development cycles. I was able to identify the important classes and make my modifications to these three classes:

Particle in particle.h :
- performs the tracking for a single streamline for a single seed, where MCMC sampling happens
Seedmanager in streamlines.h :
- manages the individual seeds, instantiates Particle objects
Counter in streamlines.h :
- keeps track of streamline coordinates in 3D-space, successful streamlines, binary brain masks, saves fiber count distributions as brain volumes

The bulk of the tracking is done using these three – the rest of the probtrackx2 code is almost entirely devoted to parsing other options and handling other input data. While I now have a lot of work to do in actually using my modifications, this foray into FSL’s source code re-emphasized three important lessons:

Documentation is critical. But not just any documentation – meaningful documentation. Even if you aren’t the best at object-oriented software development, at least describe what your code does, and give your variables meaningful names. Had their code been effectively documented, I could have been in and out of there in two or three days, but instead spent about a week figuring out what was actually going on.
You should be equally comfortable working with raw code developed by others, as you are with writing your own. Do not expect everything to be written correctly, and do not assume that just because others have used a piece of software before, that you won’t need to make modifications. Be ready to get your hands dirty.
Do not underestimate the power of compiled languages. Most data scientists work with Python and R due to the speed of development and low barrier to entry, but each is based primarily in C (and I believe not in C++ due to timing of original development cycles). Many large-scale software packages are based on languages like C, C++, and Java. Likewise, if your work bridges the gap between data scientist and engineer, you’ll definitely need to be comfortable working with compiled languages for production-level development and deployment.

Image Transformations With OpenCV

Sat, 01 Sep 2018 17:12:32 -0700

I’ve been toying around with openCV for generating MRI images with synethetic motion injected into them. I’d never used this library before, so I tested a couple examples. Below I detail a few tools that I found interesting, and that can quickly be used to generate image transformations.

# import necessary libraries
import matplotlib.pyplot as plt
import nibabel as nb
import cv2

# load image file
image_file = './data/T1w_restore_brain.nii.gz'
img_obj = nb.load(image_file)
img = img_obj.get_data()

# reorient so Anterior-Posterior axis corresponds to dim(0)
img = np.fliplr(img)
img = np.swapaxes(img, 0, 2)

# get single image slice and rescale
data = img[130, :, :]
data = (data-data.min())/data.max()
plt.imshow(data)

For any linear transformations with cv2, we can use the cv2.warpAffine method, which takes in the original image, some transformation matrix, and the size of the output image.

Let’s start with translations. The matrix will translate the image 10 pixels to the right (width), and 0 pixels down (height).

# Use the identity rotation matrix
# Third column specifies translation in corresponding direction
translation = np.array([[1, 0, 20],
 [0, 1, 0]])

translated = cv2.warpAffine(data, translation, data.T.shape)
plt.imshow(translated)

Now, in order to rotate the image, we can use cv2.getRotationMatrix2D. We’ll rotate our image by 45$^{\circ}$ .

# get shape of input image
rows, cols = data.shape

# specify angle of rotation around central pixel
M = cv2.getRotationMatrix2D((cols/2,rows/2), 45, 1)
rotated = cv2.warpAffine(data, M, (cols, rows))
plt.imshow(rotated)

Here are a few examples of randomly translating +/- 1, 5, or 9 voxels in the X and Y directions, and randomly rotating by 1, 5, or 9 degrees:

# get shape of input image
rows, cols = data.shape
# specify range of rotations and translations
txfn = [1, 5, 9]

for rt in txfn:

 # generate rotation matrix
 # randonly rotate to left or right
 M = cv2.getRotationMatrix2D(
 (cols/2, rows/2),
 np.random.choice([-1, 1], 1)[0]*rt, 1)

 # apply rotation matrix
 rotated = cv2.warpAffine(data, M, data.T.shape)

 # generate translation matrix
 # randomly translate to left or right
 T = np.array(
 [[1,0,np.random.choice([-1, 1], 1)[0]*rt],
 [0, 1,np.random.choice([-1, 1], 1)[0]*rt]]).astype(np.float32)

 # apply translation matrix
 translated = cv2.warpAffine(data, T, data.T.shape)

 # compose rotated and translated images
 movement = (rotated + translated)/2
 # compute difference between input and transformed
 difference = data-movement
 res = difference.reshape(np.product(difference.shape))

 fig,[ax1,ax2,ax3] = plt.subplots(1,3,figsize=(15,5))
 ax1.imshow(movement, cmap='gray')
 ax1.set_title('Composed Random Rotation and Translation \n Magnitude = {:}'.format(rt), fontsize=15)
 ax2.imshow(D-rotated, cmap='gray')
 ax2.set_title('Difference Map'.format(rt), fontsize=15)
 ax3.hist(res[res!=0],100,density=True)
 ax3.set_title('Difference Density', fontsize=15)
 plt.tight_layout()
 plt.show()

While this approach of generating synthetic motion into MRI images is a poor model of how motion actually occurs during an MRI scan, there are a few things I learned here. For example, if you define a measure of image similarity, like mutual information, entropy, or correlation ratio as a cost function, we can see how we can use warpAffine to find the optimal transformation matrix between two images.

I was hoping to use openCV to generate and apply 3d affine transformations to volumetric MRI data. One approach to doing this is to iteratively apply rotations and transformations along each axis – however, openCV will interpolate the data after each transformation, resulting in a greater loss of signal than I am willing to compromise on. It doesn’t seem like openCV has ability to apply 3d affine transformations to volumetric data in a single interpolation step.

A more realistic approach to generating synthetic motion artifacts that would more accurately parallell the noise-generating process, is to compute the Fast Fourier Transform of my 3d volume, and then apply phase-shifts to the k-space signal – this will also manifest as motion after applying the inverse FFT.

After doing a bit more digging through the openCV API, it seems there’s a lot of cool material for exploration – these applications specifically caught my eye and would be fun to include in projects:

video analysis for motion tracking
object recognition for detecting faces
openCV Android for app development

But alas – the search continues!

Enabling Custom Jekyll Plugins with TravisCI

Sun, 12 Aug 2018 02:14:14 -0700

I just learned about TravisCI (actually, about continuous integration (CI) in general) after attending Neurohackademy 2018. We learned about CI from the perspective of ensuring that your code builds properly when you update files in your packages, incorporate new methods, refactor your code, etc. Pretty neat.

Fast forward a couple days, and I’m trying to incorporate custom Jekyll plugins into my blog – I quickly realized GitHub doesn’t allow this for security reasons, but I couldn’t find a convenient work-around. Some posts suggested using a separate repo branch to build the site, and then push the static HTML files up to a remote repo to do the actual hosting, but for some reason I couldn’t get that approach to work.

Finally, I saw some mentions of using TravisCI and CircleCI to build and push the site using continuous integration. I ended up using the approach suggested by Josh Frankel.

Josh’s site gives a really clear explanation of the necessary steps, given some very minmal prequisite knowledge about using Git. His instructions actually worked almost perfectly for me, so I won’t repeat them again here (just follow the link above, if you’re interested) – however, there were a few issues that arose on my end:

For some reason, I had an about.html file and index.html file in the main repo directory – my built blog wouldn’t register any updates I made to about.md or index.md while these files were around, so I deleted the HTML files. This might have been an obvious bug to someone with more web programming experience, but I’m a novice at that. If you’re seeing any wonky behavior, check to make sure you don’t have any unnecessary files hanging around.
Ruby version: I had to change the version of Ruby I was using to ruby-2.4.1.
Plugins: Make sure any Jekyll plugins you want to use are already installed.
Emails: You can turn off email reporting from TravisCI by adding notifications: email: false to your .travis.yml file.

But now, you can incorporate custom, user-built Jekyll plugins and let TravisCI do the heavy lifting! I specifically wanted the ability to reference papers using BibTex-style citation links with Jekyll, like you can with LaTex or Endnote – this capability isn’t currently supported by GitHub. Happy blogging!

Rendering LaTex In Markdown Using Jekyll

Sat, 11 Aug 2018 02:14:14 -0700

In putting together this blog, I wanted to be able to talk about various mathematical topics that I found interesting, which inevitably lead to using LaTex in my posts.

I’m currently using Atom as my editor (having converted from Sublime), and needed to install a bunch of packages first. First and foremost, I wanted to be able to render my markdown posts before hosting them on the blog, and consequentially needed a way to render LaTex. For this, I installed a few Atom packages:

To preview your post in Atom, you just type ctrl+shift+M, which will display both in-line and block math sections.

However, if you build your site locally with the command bundle exec jekyll serve or push it to a remote repo, the LaTex no longer renders properly. After Googling around a bit, I determined that this was due to the way markdown converters in Jekyll, like kramdown and redcarpet, do the conversion using MathJax – specifically, in-line math segments are not properly rendered. I wanted a way to both preview the LaTex in Atom, and properly render it usng Jekyll. I found two links that solved the problem for me:

In short, the following steps solved the problem of LaTex not rendering for me. I’m using the minima theme, so I first found the theme directory with bundle show minima. In this directory, I copied the ./layouts/post.html to a local directory in my project folder called ./_layouts/post.html.

Within this file, I pasted the following two sections of HTML code:

<script type="text/x-mathjax-config">
 MathJax.Hub.Config({
 tex2jax: {
 skipTags: ['script', 'noscript', 'style', 'textarea', 'pre'],
 inlineMath: [['$','$']]
 }
 });
 </script>
<script src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>

And voila – building the posts now correctly renders LaTex!

Software Engineering |

Correlation IDs and Request Lineage Across Services

What I wanted every line to carry

How the Context Travels

Why use a filter, not a LoggerAdapter

Middleware

Grading by status

403s that explain nothing

Health probe overload

Auto-capturing tracebacks

Field names and markers

What I now get

Service-to-Service Auth with Cognito Scopes

Why a shared bearer token was not viable

Two gates

What services can call what

Cognito resource servers, custom scopes, and app clients

The resource server defines capabilities

Only custom scopes work for machine-to-machine

The app client selects a subset

Two possible gotchas

Validating a token

Fetching a token

Required ALB rules

Gotcha! Source-IP bypass breaks browser login

Gotcha! Header bypass can worsen an authentication bug

Calling it from anywhere

Wrapping up

Deploying Dagster to AWS ECS, Part 2: Pipelines

The Contract

The Two-Task-Definition Pattern

Registering With Cloud Map

TTL and Task Replacement

Four Roles Per Pipeline

The Empty Resource List Trap

Secrets Management

What I’d Do (and still might do…) Differently

Actually, One More Thing

and change the platform condition to match "run", which is the value the module actually controls.

Wrapping Up

Deploying Dagster to AWS ECS, Part 1: The Platform

Setup

Architecture overview

The read-only webserver problem

Two actions per listener rule

Finding an open listener priority

Health checks and ALB path prefixes

dagster.yaml: the compute and storage specs

Where the platform config lives

Networking: the part that took me the longest

What talks to what

Three security groups

How do tasks get their networking?

Private Subnets Need a Path to AWS APIs

What Each Failure Actually Looks Like

Service Discovery: The Namespace

IAM: Probably the crux of everything

The Shared Trust Policy

Execution Roles (Webserver + Daemon)

Daemon Task Role

PassRole Contract

What the Platform Expects From a Pipeline

Building a Schema Registry from Scratch for a Scientific Data Platform

Why not just use an existing registry tool?

Core architecture

Repository layout

Adding a schema, end to end

Schema identity: the domain/subdomain/entity pattern

Versioning: semantic, path-encoded, and strict

Export pipeline: python → YAML

Canonical dtype vocabulary

Publishing contracts

Schema templates: validating the schemas themselves

Schema metadata: queryable semantic tags

Runtime registry

API

Closing toughts

CI/CD Part 2: Docker Images and Container Registries

Multi-stage Docker builds

Create a base image

and change the platform condition to match `"run"`, which is the value the module actually controls.