Data Platform |

Correlation IDs and Request Lineage Across Services

Tue, 27 Jan 2026 09:00:00 -0800

We are a pretty small team, yet we still have a pretty large and expansive suite of services, dashboards, and backend data systems. When the number of services was small, auditting and logging was quite easy, but over the last few years, it’s become quite a chore. Imagine the following scenario:

A scientist tells you the page on the cell culture analysis dashboard was slow this morning.

If you can’t answer at least one of the following, you might find this post helpful:

Which page?
What do they mean “slow”?
Which user?
When?
What endpoint and what parameters?

The dashboard they’re referring to called some backend API, that API called two more, and one of those queried the LIMS system directly. That’s multiple services and log streams, and I’m willing to bet that not one line in any of them tells you which entries belong to that user’s click.

We face this issue in a variety of places in our data platform. I don’t mean logging in the general sense of import logging, but the narrower scope of how a request identifies itself as it moves between all of our services, so that afterwards we can put the pieces back in order and say which step was slow, or which one returned the 403 error.

I built something that I’ve been slowly deploying and incorporating into most of our services and dashboards. Three pieces do the work: a request-scoped context, middleware that fills it in, and a handful of headers forwarded on every outbound call. It pairs with the previous post on service-to-service auth, since knowing who called and when is half the battle.

What I wanted every line to carry

My goal for this tooling was that any single “typical” log line, anywhere in the system, can answer which request, which user, which service, which endpoint, how long, and what happened. This ends up being a fixed set of fields stamped onto every record key:

Field	Example	Why
`request_id`	`req-474c2493-...`	Ties every line from one request together, across services
`session_id`	`ses-a55813da-...`	Ties multiple requests from one browser session together
`user_id`	`user@example.com`	Who was actually doing this
`service`	`lims-adapter`	Which service emitted the line
`requesting_service`	`dashboard`	Which service called this one
`method` / `path`	`GET` `/samples/S-123/molecule`	Which endpoint
`client`	`10.0.5.233`	Source IP
`status_code`	`403`	What happened
`duration_s`	`1.284`	How long it took
`event`	`request_end`	Machine-readable label for the kind of line

From my perspective, request_id and session_id are the most helpful. A request ID covers one call to a service and everything it fans out into. A session ID covers everything one person did in one sitting. When someone says “it was slow this morning”, you find their session, then look at which requests inside it were slow.

How the Context Travels

There are two mechanisms for how context travels within and between requests, one inside a process and one between processes.

graph LR
Browser(Browser)
Dash[Dashboard]
ApiA[Service A]
ApiB[Service B]
Lims[LIMS]
Logs[Log store]
Browser -->|"click"| Dash
Dash -->|"X-Request-ID, X-Session-ID, X-User-ID, X-Requesting-Service: dashboard"| ApiA
ApiA -->|"same IDs, X-Requesting-Service: service-a"| ApiB
ApiA -->|"same IDs"| Lims
Dash -.->|"req-abc"| Logs
ApiA -.->|"req-abc"| Logs
ApiB -.->|"req-abc"| Logs
classDef store fill:#f3e8fd,stroke:#a142f4
class Logs store

Inside a process, the fields live in a ContextVar, so they follow the request through await chains and across threads without being passed as arguments. Between processes, they’re plain HTTP headers that outbound calls forward. X-Requesting-Service is the one that changes at each hop, while the other three carry through unchanged. That is what lets one request_id span the entire tree.

Why use a filter, not a LoggerAdapter

One way to attach context to log records is by using Python’s logging.LoggerAdapter, but I think it’s the wrong tool for what I’m trying to do. An adapter applies only to log calls made through it. Your own code gets the context, but the third-party library that raises the interesting exception does not, because it holds its own plain logging.getLogger(__name__).

A logging.Filter attached to the root handler sees every record in the process, irrespective of where it came from:

_log_context: ContextVar[dict] = ContextVar("log_context", default={})


class ContextFilter(logging.Filter):
 """Inject current context fields onto every LogRecord."""

 def filter(self, record: logging.LogRecord) -> bool:
 for key, value in {**_global_fields, **_log_context.get()}.items():
 if not hasattr(record, key):
 setattr(record, key, value)
 return True

The hasattr check means a field passed explicitly at the call site wins over the passive default context, so a caller can override user_id for one line without unbinding it. And _global_fields is a plain dict rather than a ContextVar, because ContextVar values aren’t inherited by new threads: static configuration like the service name has to live outside the context or it vanishes in a thread pool.

Binding is a context manager that nests:

@contextmanager
def log_context(**fields):
 current = _log_context.get()
 token = _log_context.set({**current, **fields})
 try:
 yield
 finally:
 _log_context.reset(token)

with log_context(request_id=rid, session_id=sid):
 log.info("request received") # two fields
 with log_context(step="validate"):
 log.info("validating") # three
 log.info("continuing") # back to two

The token mechanism restores the previous context exactly, including when the block exits by exception, which a naive save-and-restore gets wrong.

There’s also a version without a scope boundary, for values that resolve partway through a request. Identity is the usual case: you don’t know the user until auth has run, but you want every line after that point to carry it.

async def get_current_user(token: str) -> User:
 user = decode_token(token)
 update_log_context(user_id=user.email) # every later line carries it
 return user

Middleware

The middleware operates once per request: accept or generate the IDs, resolve the user, bind everything, time the request, and echo the IDs back.

request_id = request.headers.get("x-request-id") or f"req-{uuid.uuid4()}"
session_id = request.headers.get("x-session-id") or f"ses-{uuid.uuid4()}"

Accept-or-generate is what enables cross-service correlation. The first service to see a request mints the ID, and every service after that inherits the minted ID from the header. Echoing the request and session IDs back in the response headers means a client, a browser console, or a support ticket can name the exact request, which turns “it was slow this morning” into a string I can now search for in CloudWatch. Identity resolves from whichever source is available, in priority order:

x-amzn-oidc-data, the ALB-injected Cognito JWT, decoded for the email claim. This is a direct browser hit.
X-User-ID, forwarded by an upstream service that already decoded its own JWT.
A dev stub, when APP_ENV is dev, local, or unset.

The second tier carries a user’s identity into services that the user never talks to directly. Service B has no OIDC header, because its caller was service A, not a browser. It can still log which person’s click ultimately caused the call.

Grading by status

request_end is logged at ERROR for 5xx, WARNING for 4xx, and INFO otherwise:

log.log(
 logging.ERROR if status >= 500
 else logging.WARNING if status >= 400
 else logging.INFO,
 "request completed",
 extra={"event": "request_end", "status_code": status, "duration_s": duration_s},
)

403s that explain nothing

A route raises HTTPException(403, detail="missing scope: service-b/write"). The client gets a useful error. The log gets a bare 403 and nothing else. The reason is ordering: FastAPI’s ExceptionMiddleware sits inside the middleware, so by the time the middleware sees anything, the exception has already been converted into a response. The detail explaining the rejection went to the client and nowhere else. At that point, debugging a 403 means reproducing it, which is annoying and what this whole codebase is meant to fix.

The fix is an exception handler that records the reason, registered alongside the middleware:

async def http_exception_log_handler(request, exc):
 log.log(
 logging.ERROR if exc.status_code >= 500 else logging.WARNING,
 "request rejected",
 extra={
 "event": "request_rejected",
 "status_code": exc.status_code,
 "error_detail": _detail_text(exc.detail),
 },
 )
 return await _default_http_exception_handler(request, exc)

app.add_exception_handler(StarletteHTTPException, http_exception_log_handler)

It runs inside the middleware’s context, so the request, session, and user fields are already bound. It delegates to Starlette’s default handler, so the response the client receives is unchanged. And because detail is often a dict for structured errors, it gets JSON-serialized rather than str()-ed, so it stays greppable instead of turning into Python repr with single quotes.

Health probe overload

A load balancer probes /health every few seconds from every node, which means that those log lines can dominate log volume while reporting only that nothing happened, especially if they’re occuring frequently. I chose to suppress them, while still retaining useful information in the case that they fail:

Skip the request_start/request_end pair in the middleware for passing probes.
Drop uvicorn’s own access-log line for the same requests, via a filter on the root handler.

Uvicorn logs independently of your middleware, so leaving that half in place means you’ve halved the noise and kept the volume. A probe returning 4xx or 5xx is always logged – that’s the whole point of building this logging functionality. The correlation headers are still echoed on suppressed requests too, so if a probe starts failing and something retries, the retry is traceable.

Auto-capturing tracebacks

A log call made from inside an except block captures the active exception automatically, even when the caller writes log.warning("call failed") with no exc_info=True. I typically write logger.warning in error handlers constantly, and almost never remember exc_info. This results in a log full of “call failed” with no stacktrace. Capturing it by default means the traceback is there whether or not the author thought about it.

The record then carries both locations, which are usually different:

{
 "timestamp": "2026-05-26T19:09:37.015695Z",
 "level": "WARNING",
 "service": "lims-adapter",
 "message": "LIMS call failed: SQL query failed (upstream_status=405)",
 "func": "_lims_error",
 "filename": "errors.py",
 "lineno": 47,
 "request_id": "req-474c2493-18ea-42a0-be0f-5ce4d096b14c",
 "session_id": "ses-a55813da-3ab1-4471-9761-59d2ef988179",
 "user_id": "user@example.com",
 "path": "/samples/S-20260129-269/molecule",
 "exception": "Traceback (most recent call last):\n File \".../molecule.py\", line 10, in get_molecule\n ...\nLimsError: SQL query failed"
}

filename and lineno point at the error-handling code: the place someone decided to log. The traceback points somewhere else entirely, at the origin of the raise. That is where the bug lives. You want both, and they are rarely in the same file.

Field names and markers

Constants instead of string literals. A typo in fields.REQUEST_ID is a NameError at import. A typo in "request_id" is a silent schema divergence that nobody notices until a dashboard query quietly returns fewer rows than it should.

from logging_lib import fields

with log_context(**{fields.REQUEST_ID: rid, fields.USER_ID: uid}):
 log.info("handling request", extra={fields.EVENT: "request_start"})

Markers for categories. A top-level marker field tags a record for dashboards and alerting: AUDIT for user-initiated state changes, PERF for timings, SECURITY for auth and access control, SYSTEM for lifecycle, PIPELINE for job execution. It’s a small closed vocabulary, which is the point, and it makes queries trivial in whatever you’re storing logs in:

# CloudWatch Logs Insights
filter marker = "AUDIT"

# Loki
{service="lims-adapter"} | json | marker = "AUDIT"

What I now get

With the IDs in place, three questions become queries rather than investigations.

Sequencing. Filter on one request_id, sort by timestamp, and I have the ordered list of everything every service did for that one call, including the services the user never touched directly.

Lineage. requesting_service on each hop gives the call graph as it actually ran

Latency attribution. Every service records duration_s for the same request_id. The dashboard reports 3.1 seconds, service A reports 2.9, service B reports 2.7, and the LIMS query reports 2.6. The slow thing is the LIMS query, and it’s established without adding a single timer.

Service-to-Service Auth with Cognito Scopes

Tue, 09 Dec 2025 09:00:00 -0800

I’m building tools that help scientists run their day-to-day work. I work in a heavily regulated environment, subject to both U.S. and European digital policy requirements. The scenario we face regularly is: two services talk to each other. Then three. A dashboard calls a backend API, that API calls another one, a nightly job calls all of them. Somewhow, we have to answer how one service proves to another that it is allowed to make the call.

The first way I addressed this was to use Internal Bearer Tokens. Basically, you generate a long random string, store it in the AWS Secrets Manager, inject it into every service as an environment variable, have each service check inbound requests for it.

I replaced that solution with Cognito’s OAuth2 client-credentials flow. Here each service carries its own identity and each endpoint declares the scope it requires. We didn’t need to define any new infrastructure for this, since Cognito was already authenticating browser users through the ALB, so the service-to-service (S2S) path ended up reusing the same user pool, the same JWKS endpoint, and the same validation code that was already there.

Why a shared bearer token was not viable

The bearer token approach was really easy, but it wasn’t a viable approach in the long run, for a few reasons:

No expiry. A static token has no TTL. If it leaks through a log line, a memory dump, or a compromised container, it stays valid until you manually rotate it and run a redeploy of every service holding it. There is no safe window during rotation where both the old and new values work, unless you write that special-casing yourself.

No identity. When service B receives the token, it can verify the value matches what it was given, but it can’t tell who sent it. Every caller is indistinguishable, so you can only observe that the token was used.

No scope. The receiving service has no way to enforce what the caller may do. Either it accepts the token and grants full access, or it rejects it. There’s no way to express “service A may read but not write”.

Shared blast radius. The same secret works everywhere. One compromised service exposes the key to every other service, so a single breach leaves a huge hole.

Comparing these two solutions against each other, we have:

Property	Static bearer token	Cognito client credentials
Expiry	Never	~1 hour, auto-refreshed
Identity	None (opaque string)	Signed JWT with a `client_id` claim
Scope enforcement	Impossible	Per-endpoint, per-caller
Compromise blast radius	All services, indefinitely	One service, for TTL
Rotation	Manual, coordinated redeploy	Automatic
Auditability	“the token was used”	“service A called endpoint X”

Being able to attribute traffic to a caller enhances the utility of logs, which is most of the subject of my next post.

Two gates

There are two authentication and authorization gates in this architecture:

Gate 1 is the ALB. Everything in our system is either accesible from outside our VPC via the ALB, or has ingress/egress restricted to within the VPC. The ALB decides only how a request gets in: either this is an authenticated browser user who must complete an interactive Cognito login, or it’s a caller to hand straight to the service. The ALB knows nothing about your application’s permissions.

Gate 2 is the service. It validates identity and enforces scope. This is the real authorization check for S2S communication. It behaves identically no matter what language or host the caller runs on.

An incoming request has to clear both. A browser user who hasn’t been authenticated gets a 302 status to the Cognito login page via Gate 1 and never reaches the dashboard or API. A service with a valid token but the wrong scope passes Gate 1 but receives a 403 at Gate 2.

caller ──▶ [ Gate 1: ALB ] ──▶ [ Gate 2: service ] ──▶ your handler
 browser or machine? who are you, and what may you do?
 │ browser, │ no or invalid credential
 │ not logged in │ valid token, wrong scope
 ▼ ▼
 302 → Cognito login 401 / 403

What services can call what

Below is a diagram of allowed communication. Solid arrows are requests, and each one is labeled with the scope it must present. Dashed arrows are token issuance.

graph LR
Human(Browser user)
Robot(Laptop / CI job)
ALB{ALB}
Dash[Dashboard]
SvcA[Service A]
SvcB[Service B]
SvcC[Service C]
Cognito(Cognito)
Human -->|"session cookie, no scope"| ALB
Robot -->|"Authorization: Bearer"| ALB
ALB -->|"x-amzn-oidc-identity"| Dash
ALB -->|"Bearer passthrough"| SvcA
Dash -->|"service-a/read"| SvcA
Dash -->|"service-b/read"| SvcB
SvcA -->|"service-b/read"| SvcB
SvcA -->|"service-c/write"| SvcC
Cognito -.->|"token: service-a/read, service-b/read"| Dash
Cognito -.->|"token: service-b/read, service-c/write"| SvcA
classDef human fill:#e8f0fe,stroke:#4285f4
classDef machine fill:#e6f4ea,stroke:#34a853
class Human human
class Robot machine

A browser user reaches the dashboard with no scope at all, because the ALB already authenticated them interactively and the dashboard is the thing they’re allowed to look at. The dashboard’s own token is narrower than service A’s: it can read service A and service B, but nothing grants it write access to service C. If the dashboard is compromised, the write path isn’t reachable from there.

The permissions are just a table, so adding a consumer means adding a row, not changing any service’s code:

App client (the caller)	Allowed scopes
`dashboard`	`service-a/read`, `service-b/read`
`service-a`	`service-b/read`, `service-c/write`
`ci-runner`	`service-a/read`
`developer`	`service-a/read`, `service-b/read`

Cognito resource servers, custom scopes, and app clients

The resource server defines capabilities

A resource server represents a service that exposes capabilities. It has two name-like fields that do very different jobs:

# terraform code

resource "aws_cognito_resource_server" "service_b" {
 user_pool_id = var.cognito_user_pool_id

 # The scope prefix. Opaque string, conventionally a URL. Immutable in practice.
 identifier = "https://api.example.com/service-b"

 # Console display name only. Nothing references it.
 name = "service-b"

 scope {
 scope_name = "read"
 scope_description = "Read access to service-b"
 }

 scope {
 scope_name = "write"
 scope_description = "Write access to service-b"
 }

 scope {
 scope_name = "admin"
 scope_description = "Administrative access to service-b"
 }
}

The identifier field becomes the prefix of every scope the resource server defines, so the three scopes above are really:

https://api.example.com/service-b/read
https://api.example.com/service-b/write
https://api.example.com/service-b/admin

That full string is what a client requests, what appears in the JWT’s scope claim, and what a service compares against. name is a display label in the console and is referenced by nothing.

Only custom scopes work for machine-to-machine

The client-credentials flow works only with custom scopes from a resource server. I defined these scope manually in my FastAPI applications, and decorated the relevant endpoints with their respective scopes. The built-in OpenID scopes (openid, email, profile, aws.cognito.signin.user.admin) are for user-facing flows, and asking for one with grant_type=client_credentials fails. Every machine caller needs at least one resource server to exist, even if the service it calls has exactly one capability.

The app client also needs a secret and the flow explicitly enabled. A client without generate_secret cannot do client credentials at all, and the user pool needs a domain configured, since the token endpoint lives at https://<domain>.auth.<region>.amazoncognito.com/oauth2/token.

The app client selects a subset

An app client represents a caller of a service. One per calling service, with allowed_oauth_scopes listing every downstream capability it may request:

resource "aws_cognito_user_pool_client" "service_a_m2m" {
 name = "service-a-m2m"
 user_pool_id = var.cognito_user_pool_id

 generate_secret = true
 allowed_oauth_flows_user_pool_client = true
 allowed_oauth_flows = ["client_credentials"]

 # Full scope strings, exactly as composed above.
 allowed_oauth_scopes = [
 "https://api.example.com/service-b/read",
 "https://api.example.com/service-c/write",
 ]

 # Cognito validates that each scope exists, so the resource servers
 # must be created first. Terraform will not infer this ordering.
 depends_on = [
 aws_cognito_resource_server.service_b,
 aws_cognito_resource_server.service_c,
 ]
}

Service A is granted read on service B, and write on service C. It cannot write to service B even though that scope exists, because its app client never lists it as a viable permissions. Cognito rejects an app client referencing a scope that doesn’t exist yet, and Terraform sees no dependency between the two resources because the scope is a hand-written string rather than a reference to the resource server’s attributes.

Scopes model capabilities, not consumers. Service B declares that reading, writing, and administering are things that can be done to it (this is defined both in the FastAPI code via decorated endpoints, and in the Cognito resource server). But Service B doesn’t say anything about who can take those actions. Consumers get assigned action permissions through their app client’s allowed-scopes list.

That separation lets you onboard a caller without touching the service being called. A new consumer is one app client and one secret. Service B does not redeploy, does not learn the new caller’s name, and does not grow a config entry. Its code already says require_scope("read") and will keep saying that no matter how many services eventually call it.

Two possible gotchas

An omitted scope parameter is not the same as no scopes. If a token request leaves scope off entirely, Cognito issues a token carrying every scope the app client is allowed. This means that every downstream call then presents maximum privilege. Always request scopes explicitly, one per downstream call. This is also why caching tokens per scope rather than per client matters.

Removing a scope is harder than adding one. Cognito will not let you delete a scope from a resource server while an app client still lists it in allowed_oauth_scopes. The order is: remove it from every app client, apply, then remove it from the resource server. In Terraform that’s two applies, and doing it in one produces an error that names the resource server rather than the app client that’s actually the problem.

Validating a token

The receiving side has to check four things:

The signature, against Cognito’s public JWKS for the pool.
The issuer, so a token from some other pool isn’t accepted.
token_use == "access", so an ID token can’t be substituted for an access token.
The scope claim contains the specific scope the endpoint requires.

def _decode_token(self, token: str) -> dict:
 signing_key = self._jwk_client.get_signing_key_from_jwt(token)

 claims = jwt.decode(
 token,
 signing_key.key,
 algorithms=["RS256"],
 issuer=self.issuer,
 # Client-credentials tokens carry `client_id` instead of `aud`,
 # so skip audience verification and check token_use instead.
 options={"verify_aud": False},
 )

 if claims.get("token_use") != "access":
 raise HTTPException(401, "wrong token type")

 return claims

Client-credentials tokens have no aud claim. Instead, they carry client_id , so leaving audience verification on rejects every valid S2S token with a confusing error. Turning it off here is valid, but only because token_use and the issuer are being checked instead.

Scope enforcement then becomes a dependency on the router:

full_scope = f"{self.resource_server_id}/{self._scope}"
if full_scope not in claims.get("scope", "").split():
 raise HTTPException(403, f"token missing required scope: {full_scope}")

app = FastAPI(dependencies=[Depends(auth)]) # authenticated everywhere
router = APIRouter(dependencies=[Depends(auth.require_scope("read"))]) # plus a scope

There is also a design decision buried here. Browser requests arriving with ALB OIDC headers pass the scope check without a scope, since the ALB already authenticated the human interactively and browser sessions have no scopes to check.

Fetching a token

The outbound half fetches tokens from Cognito and caches them until they expire.

Cache per scope, not per client. A service calling three downstream APIs holds three tokens, each carrying only the access it needs for that call.

Guard the refresh with a lock, and re-check inside it. Under concurrent load, several coroutines will notice the expired token at the same moment and all stampede Cognito’s token endpoint:

async def get_token(self, scope: str) -> str:
 cached = self._tokens.get(scope)
 if cached and time.time() < cached.expires_at:
 return cached.access_token

 if scope not in self._locks:
 self._locks[scope] = asyncio.Lock()

 async with self._locks[scope]:
 # Re-check inside the lock: another coroutine may have fetched already.
 cached = self._tokens.get(scope)
 if cached and time.time() < cached.expires_at:
 return cached.access_token
 self._tokens[scope] = await self._fetch_new_token(scope)
 return self._tokens[scope].access_token

Expiry is stored with a 60 second skew subtracted, so a token is treated as expired slightly before Cognito thinks so. Without that, a token that passes the check and then spends 400ms in flight can arrive already invalid.

Required ALB rules

If your ALB has a Cognito authentication action on its listener rule, it applies to every request on that rule. A service presenting a valid Bearer token gets a 302 redirect to the Cognito login page. It never reaches your service, so all that careful token validation never runs.

The listener needs to rules, defined in order:

Higher priority: match requests carrying an Authorization: Bearer header and forward them straight to the EC2 target group, with no authenticate action.
Lower priority: everything else gets authenticate-cognito and the normal browser login.

With respect to Gate 1, you could also consider matching on source IP instead, forwarding anything from an allowlisted CIDR. This fixes Gate 1, and Gate 2 is unchanged either way.

However, a source-IP matching rule forwards without running the authenticate action. So once you allowlist the office or VPN range, a person opening a dashboard in a browser from that network skips the login entirely and arrives with no session and no token. This is DEFINITELY NOT what you want, because we want all requests to be authenticated. This is specific to browser users, and S2S communication still works.

Also, allowlisting the VPC’s own CIDR admits in-VPC services only. A public ALB sees a laptop’s corporate or VPN egress IP. That address is nowhere near the VPC range. So a VPC-CIDR rule never admits laptops, even on VPN. If in-VPC calls work and laptops get redirected to a login page, this is why.

Gotcha! Header bypass can worsen an authentication bug

We trust browser requests based on the presence of the x-amzn-oidc-identity header, and that the browser path skips scope checks. That’s normally safe, because the ALB’s Cognito action overwrites that header on every request it processes, so a client can’t forge it.

A bypassed request doesn’t run that action, so with a header-match rule, anyone on the internet can send:

Authorization: Bearer anything
x-amzn-oidc-identity: someone@example.com

The first header triggers the bypass. The second makes your service treat the request as an authenticated browser user, which skips the scope check entirely. The Bearer value is never validated, because the ALB-header path returns before the token path is reached.

The mitigations, either of which closes it:

Strip inbound x-amzn-oidc-* headers on the bypass path, so only the ALB can ever set them.
Dont trusting identity by presence of the header alone. Verify the signed x-amzn-oidc-data JWT rather than reading x-amzn-oidc-identity as a plain string.

The second is the better fix because it removes a whole class of bugs instance of one instance of it.

Calling it from anywhere

Something that confused me initially was that curl requests from the CLI would fail. But this was because we weren’t passing in any form of authentication into the request, which made local testing or scenarios like calling a service directly from an EC2 instance, fail. I enabled this functionality with two HTTP steps that are pretty much identical in Python, Java, or a shell script:

TOKEN=$(curl -s -X POST "https://<domain>.auth.<region>.amazoncognito.com/oauth2/token" \
 -u "$CLIENT_ID:$CLIENT_SECRET" \
 -d grant_type=client_credentials \
 -d 'scope=https://api.example.com/service-a/read' | jq -r .access_token)

curl "https://api.example.com/service-a/things" -H "Authorization: Bearer $TOKEN"

A Python client is a convenience wrapper around these steps – it’s not the ONLY way to run this authorization flow. Anything that can make an HTTPS request can participate. But we might want to alleviate this by building in a seperate app client for developers and for automation purposes.

Wrapping up

This was a pretty approachable problem, especially since we already had the infrastructure in place. We were already using Cognito to authenticate browser users through the ALB, and the client-credentials flow runs against the same user pool, the same JWKS endpoint, and the same validation code. We just needed one more OAuth flow on the system already in place.

Deploying Dagster to AWS ECS, Part 2: Pipelines

Wed, 05 Nov 2025 09:00:00 -0800

Here is the second of two posts on running Dagster in ECS. The first dealt with the platform side: the Daemon, the two Webservers, the ALB and Cognito wiring, the three security groups, the Cloud Map namespace, and the IAM that lets the Daemon launch anything at all.

The first component I referred to as dagster-platform. This one is about what I’m referring to as dagster-pipeline, the module you instantiate once per pipeline. Platform infrastructure gets deployed infrequently. Pipelines get deployed several times a day by whoever happens to be working on them. The reason I separated the two is that pushing a new job should never touch a security group, an IAM policy, or a load balancer rule.

The Contract

Part One ended with the four things the platform expects from a pipeline:

Register a service in the Cloud Map namespace, so the Daemon and Webserver can resolve it by DNS on port 4000.
Attach the usercode service to the platform’s usercode security group, so that traffic is actually allowed.
Tag its run roles with dagster:component and dagster:managed-by, so the Daemon is permitted to pass them to ECS.
Get its code location into workspace.yaml in SSM, so the Webserver and Daemon know it exists.

The platform defines the security group and the pipeline looks it up by name:

# dagster-pipeline/ecs.tf
data "aws_security_group" "usercode" {
 name = "${var.usercode_sg}-${var.platform_env}"
}

Here is the whole module and everything it touches. Blue is the platform, which this module does not create and only attaches to. Green is what the pipeline module builds.

graph LR
Daemon[daemon]
Web[webserver]
UcSvc[usercode service]
RunTD(run task def)
Run[run task]
CM(Cloud Map)
ECR(ECR)
SM(Secrets Manager)
PG(Postgres)
S3(S3)
SrcDB(source DB)
Web -->|4000| UcSvc
Daemon -->|4000| UcSvc
UcSvc -.->|register| CM
CM -.->|resolve| Daemon
UcSvc -.->|names| RunTD
Daemon -->|RunTask| RunTD
RunTD -.->|instantiates| Run
ECR -.->|image| UcSvc
ECR -.->|image| Run
SM -->|exec role| UcSvc
SM -->|task role| Run
UcSvc -->|5432| PG
Run -->|5432| PG
Run -->|443| S3
Run -->|query| SrcDB
classDef platform fill:#e8f0fe,stroke:#4285f4
classDef pipeline fill:#e6f4ea,stroke:#34a853
classDef ephemeral fill:#fef7e0,stroke:#f9ab00
classDef backend fill:#f1f3f4,stroke:#9aa0a6
class Daemon,Web platform
class UcSvc,CM,RunTD pipeline
class Run ephemeral
class ECR,SM,PG,S3,SrcDB backend

The run task definition is a definition, not a running thing: the usercode container names it in an environment variable, and the daemon is what actually instantiates it. The arrows pointing into the Secrets Manager land on different roles, because the usercode service has its credentials injected by ECS before it starts while the run task reads them itself at runtime.

The Two-Task-Definition Pattern

Each pipeline module creates two task definitions:

usercode task definition: the always-on code server. This runs dagster code-server start and stays alive, serving job definitions, sensors, schedules, and so on to the Daemon. It registers with Cloud Map so the platform can discover it.

run task definition: the task that executes a job. When a run is triggered, the EcsRunLauncher spins up a new Fargate task using this definition. It’s ephemeral, meaning it starts, runs the job, and exits.

The two definitions point at the same image and differ mainly in CPU/memory and in which IAM roles they use. The run definition’s command doesn’t really matter, since the launcher overrides it with the actual run command, but I set it to the same code-server command so that starting the task by hand does something reasonable. The usercode container needs to know about the run task definition so it can tell Dagster which task to launch, which can be achieved with two environment variables.

locals {
 dagster_current_image_env = [
 { name = "DAGSTER_CURRENT_IMAGE", value = data.aws_ecr_image.usercode.image_uri }
 ]

 dagster_container_context = {
 ecs = {
 task_definition_arn = aws_ecs_task_definition.run.arn
 container_name = "run"
 }
 }

 dagster_container_context_env = [
 { name = "DAGSTER_CONTAINER_CONTEXT", value = jsonencode(local.dagster_container_context) }
 ]
}

DAGSTER_CURRENT_IMAGE is the pipeline’s ECR image URI. DAGSTER_CONTAINER_CONTEXT tells Dagster which ECS task definition and container name to use for runs. Note that container_name has to match the name field of the container in the run task definition exactly. If it doesn’t, the run task launches and Dagster can’t find the container it’s supposed to be watching. Both of these go on the usercode container only, not the run container. Building DAGSTER_CONTAINER_CONTEXT from aws_ecs_task_definition.run.arn also creates the dependency that forces Terraform to build the run task definition first.

Registering With Cloud Map

The platform created the private DNS namespace, and each pipeline creates its own service record inside of that DNS namespace:

# dagster-pipeline/cloud_map.tf

# Look up the namespace created by the platform module
data "aws_service_discovery_dns_namespace" "dagster" {
 type = "DNS_PRIVATE"
 name = "pipelines-${var.env}.usercode"
}

# Each pipeline gets its own service record in the shared namespace
resource "aws_service_discovery_service" "usercode" {
 name = "${var.pipeline_name}-pipeline-${var.env}"

 dns_config {
 namespace_id = data.aws_service_discovery_dns_namespace.dagster.id

 dns_records {
 ttl = 300
 type = "A"
 }

 routing_policy = "MULTIVALUE"
 }
}

The ECS service ties itself to that record with a service_registries block, and from then on ECS registers the Fargate task’s private IP with Cloud Map automatically:

resource "aws_ecs_service" "usercode" {
 # ...
 service_registries {
 registry_arn = aws_service_discovery_service.usercode.arn
 }
}

The DNS name becomes <pipeline-name>-pipeline-<env>.pipelines-<env>.usercode, which goes into workspace.yaml:

# workspace.yaml (stored in SSM)
load_from:
 - grpc_server:
 host: ingest-pipeline-prod.pipelines-prod.usercode
 port: 4000
 location_name: "ingest"
 - grpc_server:
 host: transforms-pipeline-prod.pipelines-prod.usercode
 port: 4000
 location_name: "transforms"

“ingest” and “transforms” are what you see in the Dagster webserver when looking at each of your pipeline deployments.

TTL and Task Replacement

The ttl = 300 in that dns_records is important here. Fargate tasks get a new private IP every time they’re replaced, and a code server is replaced on every deploy. Cloud Map updates the A record immediately, but the Daemon and Webserver are resolving that name through the VPC resolver, which respects the TTL. For a few minutes after a deploy, they can be holding the IP of a task that no longer exists, and in practice, that means that a code location goes red (a.k.a looks like a failed deployment in the UI) in the UI right after a deploy and then fixes itself a few minutes later. Decreasing the TTL to 15 or 30 seconds makes deploys settle much faster, at the cost of more DNS queries, which for a handful of code servers is probably acceptable.

routing_policy = "MULTIVALUE" returns every healthy instance registered under the name. With desired_count = 1 there’s only ever one, but it’s the right policy if you later run more than one replica of a code server. Similarly, when you add a new pipeline, you need to update workspace.yaml in SSM and restart the Daemon and Webserver services so they pick up the new entry. I’m currently running this manually, but it’s worth automating in the future (but not a deal-breaker, just not something I’d consider “complete”).

Four Roles Per Pipeline

Each pipeline module creates four IAM roles: execution and task roles for both the usercode service (always-on code server) and the run task (ephemeral, launched per job execution).

Just like I mentioned in Part One, if you’re handling AWS secrets, if the value is injected by the ECS secrets block in the task definition, the execution role needs to read it. If your application code calls boto3 to fetch it, the task role needs to read it.

The roles are structurally identical, but the tags differ. The tags are what the platform’s iam:PassRole condition checks against:

# dagster-pipeline/iam.tf

locals {
 usercode_role_tags = {
 "dagster:component" = "usercode"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
 }

 run_role_tags = {
 "dagster:component" = "run"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
 }
}

# Run execution role, used by ECS to start the ephemeral run container
resource "aws_iam_role" "run_exec" {
 name = "${var.name_prefix}-${var.pipeline_name}-run-exec-role-${var.pipeline_env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
 tags = merge(local.run_role_tags, var.tags)
}

resource "aws_iam_role_policy_attachment" "run_exec_attach" {
 role = aws_iam_role.run_exec.name
 policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Run task role, used by the running job container to call AWS APIs
resource "aws_iam_role" "run_task" {
 name = "${var.name_prefix}-${var.pipeline_name}-run-task-role-${var.pipeline_env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
 tags = merge(local.run_role_tags, var.tags)
}

The Empty Resource List Trap

IAM rejects a policy statement with an empty Resource list, so a pipeline that connects to no databases can’t just get a policy document with zero ARNs in it. The document has to not exist at all:

data "aws_iam_policy_document" "db_secrets_read" {
 # A pipeline that reads no databases must not get this policy at all.
 count = length(var.db_secret_ids) > 0 ? 1 : 0

 statement {
 sid = "ReadDatabaseSecrets"
 effect = "Allow"
 actions = ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"]
 resources = [for s in data.aws_secretsmanager_secret.db_secrets : s.arn]
 }
}

resource "aws_iam_role_policy" "run_db_secrets" {
 count = length(var.db_secret_ids) > 0 ? 1 : 0
 name = "${var.name_prefix}-${var.pipeline_name}-run-db-secrets-${var.pipeline_env}"
 role = aws_iam_role.run_task.id
 policy = data.aws_iam_policy_document.db_secrets_read[0].json
}

Secrets Management

I’m using two different categories of secrets. Dagster’s own Postgres credentials are stored as a single Secrets Manager secret with JSON keys. These get injected as individual environment variables using the JSON key syntax in the ECS task definition secrets block:

locals {
 dagster_container_secrets = [
 {
 name = "DAGSTER_POSTGRES_HOSTNAME"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:hostname::"
 },
 {
 name = "DAGSTER_POSTGRES_USER"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:username::"
 },
 {
 name = "DAGSTER_POSTGRES_PASSWORD"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:password::"
 },
 {
 name = "DAGSTER_POSTGRES_DB"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:name::"
 }
 ]
}

The :key:: suffix trailing fields are the version stage and version ID. When those are empty, you get the most recent / current version. Since these are injected by ECS, the execution role is what needs GetSecretValue on the secret.

Source database credentials (for pipelines that read from other databases) are not injected at all anymore. My first version of the Terraform module wrote a .pgpass file at container startup: each secret ID came in as PGPASS_SECRET_JSON_1, PGPASS_SECRET_JSON_2 and so on, plus a PGPASS_SECRET_JSON_VARS variable listing which env vars to look at, and the entrypoint fetched each one and assembled the file. It worked, but it was messy and became really annoying to manage since I was dragging around another script for building the .pgpass file. It also meant credentials sat in the container environment and on disk for the life of the task, the entrypoint had to grow a chunk of logic that had nothing to do with running jobs, so I replaced it with the following

variable "db_secret_ids" {
 type = list(string)
 description = "Secrets Manager secrets for the databases this pipeline connects to. The task role is granted GetSecretValue on them; the application reads them at runtime."
 default = []
}

import json
import boto3


def get_connection_details(secret_id: str) -> dict:
 client = boto3.client("secretsmanager")
 secret = client.get_secret_value(SecretId=secret_id)
 return json.loads(secret["SecretString"])

The credentials are never in the environment and never on disk in the task, and adding a database is one more entry in a list variable.

What I’d Do (and still might do…) Differently

Automate workspace.yaml updates. Right now, adding a pipeline requires a manual SSM update and service restart. A Lambda triggered by ECS task state changes, or even a simple CI step, could handle this.

Use Cognito + proper OIDC federation for access control. The dual-webserver approach works, but it’s not pretty. If you control your IdP, set up Cognito federation from the start. I tried setting up a proxy server to redirect traffic based on users belonging to one Cognito user-group or another, but this ended up being a huge headache.

Consider ECS Exec for debugging. I didn’t enable it initially and spent a lot of time reading logs trying to diagnose container startup issues. aws ecs execute-command is worth the extra IAM policy on the task role.

Don’t put dagster.yaml in the platform images. Storing it in SSM and injecting at runtime was the right call for the daemon and webserver. Rebuilding (EVERY) image every time you want to change a config is not the move. For pipeline images, where the code changes anyway, baking it in is fine and simpler.

Fetch application secrets at runtime instead of injecting them. See above. This is the change I’m happiest about.

Actually, One More Thing

As I was writing up this post, I went back to the IAM code to check that I’d described the tag conditions correctly, and found a bug.

Here’s the setup again. The platform’s iam:PassRole policy only allows the Daemon to pass roles tagged dagster:component = "pipeline". The pipeline module’s local says something else:

run_role_tags = {
 "dagster:component" = "run" # not "pipeline"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
}

"run", not "pipeline". By that reading the Daemon should never be able to pass these roles, and every run should fail. But runs work fine in both environments, so I had never noticed. The correct approach is in the IaC merge command

tags = merge(local.run_role_tags, var.tags)

merge lets later arguments win, so var.tags overrides the module’s own local. And every caller passes a tag block that happens to include exactly the key in question:

tags = {
 "env" = var.pipeline_env
 "owner" = "data-platform"
 "application" = "dagster"
 "dagster:pipeline" = var.pipeline_name
 "dagster:component" = "pipeline" # this is what makes PassRole work
 "dagster:managed-by" = "terraform"
}

So the roles do come out tagged with pipeline, and the security control does work, but only because of the argument order in a merge and a tag the caller happens to set. A new pipeline that passes a tags map without dagster:component would plan clean, apply clean, start its code server, show up in the UI, and then fail the first time someone launched a run, with an AccessDenied on PassRole and nothing in the Terraform to suggest why.

The fix is to stop letting callers override the tag the policy depends on:

tags = merge(var.tags, local.run_role_tags)

and change the platform condition to match `"run"`, which is the value the module actually controls.

Wrapping Up

Getting Dagster onto ECS was genuinely hard for me. The documentation covers the “in a perfect world” scenario, but not the real-world complexity of integrating with existing cloud infrastructure that you don’t partially (or entirely) control. There are plenty of “in a perfect world with a clean slate, here’s how you do it” walkthroughs, but very little in the way of justification or documentation of architecture choices. The platform/pipeline split in Terraform has been a huge win for me. I’ve templated our Dagster pipeline development platform and put together templated Dockerfiles and docker-compose.yaml files. Deploying a new pipeline is now a terraform apply with a handful of variables, no platform infrastructure touched, no service restarts required (except for the workspace.yaml update, which I’m working on).

Deploying Dagster to AWS ECS, Part 1: The Platform

Tue, 04 Nov 2025 09:00:00 -0800

Here at Just-Evotec Biologics, the Data Platform team uses Dagster as our orchestration platform of choice. Many of our services are deployed to AWS ECS as Fargate or EC2 tasks. Getting Dagster onto ECS took me an embarrassing number of hours, and most of those issues stemmed from a lack of documentation around getting Dagster pipelines up and running in the cloud while NOT using Dagster+ (their premium service offering). The official docs cover the happy, perfect world path, but beyond that, you’re left stitching together forum threads, GitHub issues, and a lot of trial and error. I figured I’d put together some notes on what I built, and how I built it.

The components are of this system are two-fold – the actual Python-based pipeline code, and the Terraform code I used to deploy that code to ECS. The Terraform part also comes in two pieces, because I’ve split the deployment down the middle. In this post, I’ll detail the platform half: the long-lived infrastructure everything else plugs into, such as the webserver and the daemon. Part two covers the pipeline module, what you’d instantiate per pipeline and then redeploy repeatedly as you make updates to the pipeline code.

If you came here because something in your deployment cannot reach something else, it might be helpful for you to skip straight to the networking section, covering security groups, ports, service discovery. But if not, read on…

Setup

Our team already had a multi-environment AWS setup, with VPCs, subnets, security groups, tunneling, and load balancers in place. The goal was to plug Dagster into that existing infrastructure as cleanly as possible, with proper separation of concerns. Again, as I mentioned in some other posts, I obviously can’t share our company code, so I’ll just describe some of the approaches I took.

I split the Terraform code into two distinct modules:

dagster-platform: the long-lived infrastructure. Daemon, Webserver(s), shared IAM roles, ALB listener rules, SSM parameters, Cloud Map namespace, and S3/RDS backends.
dagster-pipeline: one instantiation per pipeline. Each pipeline gets its own ECS task definitions, service, Cloud Map entry, IAM roles, and log groups. You call this module once per pipeline and it slots right into the platform.

This split makes sense from a frequency-of-interaction standpoint. The platform piece deploys infrequently, and doesn’t need to be modified much. Pipelines deploy constantly, sometimes multiple times a day, and I don’t want to touch platform infrastructure every time I push a new job.

Architecture overview

Here’s the high-level picture:

ALB (existing)
 ├── /pipelines/admin/* → Webserver (read-write) [Cognito-authenticated]
 └── /pipelines/* → Webserver (read-only) [Cognito-authenticated]

Supporting Services
 ├── RDS PostgreSQL (run storage, event log, schedule storage)
 ├── S3 (compute logs)
 ├── AWS Secrets Manager (DB credentials, application secrets)
 ├── SSM Parameter Store (dagster.yaml, workspace.yaml)
 └── Cloud Map (service discovery for usercode servers)

ECS Cluster (existing)
 ├── dagster-daemon (always-on Fargate service)
 ├── dagster-webserver-rw (always-on Fargate service)
 ├── dagster-webserver-ro (always-on Fargate service)
 └── pipeline-<name> (always-on usercode service, one per pipeline)

Everything above the pipeline-<name> line is the platform – that’s what this current post is about. The Daemon and Webserver containers share the same dagster.yaml and workspace.yaml, loaded at runtime from SSM. Pipelines register themselves via Cloud Map so the Daemon can find them.

The read-only webserver problem

Dagster’s dagster-webserver supports a --read-only flag, but there’s no built-in authentication layer that lets you say “scientists get read-only access, data engineers get admin access.” Dagster does have a subscription tier called Dagster+ that does offer this functionality, but we do not pay for that. So, we’re lefting putting our own solution in place. I’m not sure if there is a “right” answer here. One solution, given our constraints, is to combine Cognito with our IdP federation, but since I don’t control our company’s IdP tenant, I couldn’t get that working. Instead, I deployed two separate ECS services, one read-write and one read-only, each behind its own ALB target group and listener rule.

Read-write webserver command:

dagster-webserver -h 0.0.0.0 -p 3000 \
 -w /opt/dagster/dagster_home/workspace.yaml \
 -l /pipelines/admin

Read-only webserver command:

dagster-webserver -h 0.0.0.0 -p 3000 \
 -w /opt/dagster/dagster_home/workspace.yaml \
 -l /pipelines \
 --read-only

The -l flag sets the URL prefix. The ALB rules do the routing:

/pipelines/admin* → read-write target group (higher priority)
/pipelines* → read-only target group (lower priority)

Both rules are Cognito-authenticated, so anyone hitting either URL has to log in. Scientists get the read-only URL. Engineers get the admin URL. Simple, but it works.

Two actions per listener rule

An authenticated listener rule is two actions in order. The authenticate-cognito action runs first, then the forward action sends the request to the target group. If you only write the forward action, the rule works and nobody has to log in, which is the kind of mistake you don’t notice until someone tells you they never saw a login screen.

resource "aws_lb_listener_rule" "pipelines_admin_rw" {
 listener_arn = data.aws_alb_listener.webserver.arn
 priority = local.admin_listener_priority

 action {
 type = "authenticate-cognito"
 authenticate_cognito {
 user_pool_arn = data.aws_cognito_user_pool.dagster.arn
 user_pool_client_id = data.aws_cognito_user_pool_client.dagster.id
 user_pool_domain = data.aws_cognito_user_pool.dagster.domain

 on_unauthenticated_request = "authenticate"
 scope = "openid"
 session_timeout = 3600
 }
 }

 action {
 type = "forward"
 target_group_arn = aws_lb_target_group.webserver_rw.arn
 }

 condition {
 path_pattern {
 values = ["/${var.alb_rule_listener_rule_path}/admin*"]
 }
 }
}

The read-only rule is identical except for the path pattern and target group, plus a depends_on pointing at the admin rule so the two are never created in the wrong order.

Finding an open listener priority

ALB listener rule priorities have to be unique across the whole listener, and they need to be specified. Neither Terraform nor AWS will “infer” the next open value. Our ALB is shared with other applications, so hardcoding numbers means constantly checking what’s available in the console or through the CLI. I ended up writing a small bash script that the external data source calls at terraform plan time. It lists the existing rules and returns the first N unused priority slots:

data "external" "listener_priorities" {
 program = ["bash", "${path.module}/external/next_listener_priorities.sh"]

 query = {
 listener_arn = data.aws_alb_listener.webserver.arn
 region = data.aws_region.current.id
 profile = var.aws_profile
 count = 2
 }
}

locals {
 admin_listener_priority = tonumber(data.external.listener_priorities.result.priority_1)
 ro_listener_priority = tonumber(data.external.listener_priorities.result.priority_2)
}

And the script itself, which is a jq expression over describe-rules:

#!/usr/bin/env bash
set -euo pipefail

INPUT_JSON="$(cat)"
LISTENER_ARN="$(jq -r '.listener_arn' <<<"$INPUT_JSON")"
REGION="$(jq -r '.region' <<<"$INPUT_JSON")"
COUNT="$(jq -r '.count // 2' <<<"$INPUT_JSON")"

RULES_JSON="$(aws --region "$REGION" elbv2 describe-rules \
 --listener-arn "$LISTENER_ARN" --output json)"

# Find the first COUNT missing priorities in [1..50000]
jq -c --argjson count "$COUNT" '
 [ .Rules[].Priority | select(. != "default") | tonumber ] | unique as $used
 | reduce range(1; 50001) as $i
 ({out: []};
 if (.out | length) >= $count then .
 elif ($used | index($i)) == null then .out += [$i]
 else . end
 )
 | .out as $m
 | reduce range(0; ($m | length)) as $idx
 ({}; . + {("priority_" + (($idx + 1) | tostring)): ($m[$idx] | tostring)})
' <<<"$RULES_JSON"

The external data source runs on every plan, so the machine running Terraform needs the AWS CLI and jq installed and credentials that can call describe-rules (this could be your local machine, your CI/CD runner, an EC2 instance, etc.) The priorities are read at plan time, so if someone else claims a slot between your plan and your apply, the apply fails, though that’s rare enough that I’ve been willing to live with it. Basically, hardcoding priorities was not an option I wanted to entertain.

Health checks and ALB path prefixes

The ALB health checks for the webservers need to use the correct path prefix. If your webserver is mounted at /pipelines and your health check pings /, it’ll get a redirect (or a 404) and the target will never go healthy.

Read-only health check path: /<your-prefix> Read-write health check path: /<your-prefix>/admin

health_check {
 protocol = "HTTP"
 path = "/${var.alb_rule_listener_rule_path}/admin"
 matcher = "200-399"
 healthy_threshold = 2
 unhealthy_threshold = 3
 interval = 30
 timeout = 25
}

Set the matcher to 200-399 to handle redirects during startup. The Dagster webserver takes 15 to 30 seconds to come up on a cold start, so unhealthy_threshold and interval need to be generous enough that a slow container start doesn’t put the service into a restart loop.

dagster.yaml: the compute and storage specs

The dagster.yaml configuration basically let’s you define job queing, which system will run your jobs (here, ECS), where logs and compute history will be stored, etc. Here’s what our final config looks like (pretty similar to what’s already out there):

run_coordinator:
 module: dagster.core.run_coordinator
 class: QueuedRunCoordinator
 config:
 max_concurrent_runs: 15

scheduler:
 module: dagster.core.scheduler
 class: DagsterDaemonScheduler

run_launcher:
 module: dagster_aws.ecs
 class: EcsRunLauncher
 config:
 use_current_ecs_task_config: true
 include_sidecars: true
 run_task_kwargs:
 cluster: "your-ecs-cluster-name"

compute_logs:
 module: dagster_aws.s3.compute_log_manager
 class: S3ComputeLogManager
 config:
 bucket:
 env: DAGSTER_LOG_BUCKET
 prefix: "compute_logs"
 region: "your-aws-region"

run_storage:
 module: dagster_postgres.run_storage
 class: PostgresRunStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

schedule_storage:
 module: dagster_postgres.schedule_storage
 class: PostgresScheduleStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

event_log_storage:
 module: dagster_postgres.event_log
 class: PostgresEventLogStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

retention:
 schedule:
 purge_after_days: 90
 sensor:
 purge_after_days:
 skipped: 7
 failure: 30
 success: -1

The ECS task configuration was confusing to me initially. use_current_ecs_task_config: true tells the EcsRunLauncher to take the network configuration (subnets, security groups, cluster) from the task that requested the run rather than making you restate all of it in run_task_kwargs. Combined with a per-pipeline run task definition, which is covered in part two, it means each pipeline’s runs launch into the same networking as that pipeline’s code server, with that pipeline’s own IAM roles. You don’t need to define a shared task definition across pipelines, which keeps the principle of least privilege in play.

Every credential in that file is an env: reference, and nothing secret is in the file itself.

Where the platform config lives

There are two config files: dagster.yaml (instance config, above) and workspace.yaml (the list of code locations the Daemon and Webserver load over gRPC). Baking config into the Docker image means rebuilding and redeploying the image every time you tweak a config value. For the Daemon and Webserver that’s a bad idea, because those images almost never change, but the config does. So Terraform writes both files into SSM parameters:

resource "aws_ssm_parameter" "dagster_yaml" {
 name = "${var.ssm_prefix}/dagster.yaml"
 type = "String"
 value = var.dagster_yaml
}

resource "aws_ssm_parameter" "workspace_yaml" {
 name = "${var.ssm_prefix}/workspace-${var.platform_env}.yaml"
 type = "String"
 value = var.workspace_yaml
}

and the task definitions inject them as container secrets:

locals {
 dagster_config_ssm_secrets = [
 { name = "DAGSTER_YAML", valueFrom = aws_ssm_parameter.dagster_yaml.arn },
 { name = "WORKSPACE_YAML", valueFrom = aws_ssm_parameter.workspace_yaml.arn },
 ]

 common_platform_env = [
 { name = "DAGSTER_CONFIG_FROM_ENV", value = "1" },
 { name = "DAGSTER_DEPLOYMENT_ENV", value = var.platform_env },
 { name = "DAGSTER_HOME", value = var.dagster_home },
 { name = "DAGSTER_LOG_BUCKET", value = "${var.dagster_log_bucket}-${var.platform_env}" },
 ]
}

DAGSTER_CONFIG_FROM_ENV is a flag the entrypoint script checks. When it’s set, the script writes $DAGSTER_YAML and $WORKSPACE_YAML out to $DAGSTER_HOME/dagster.yaml and $DAGSTER_HOME/workspace.yaml before starting the process, and leaves whatever is in the image alone otherwise. Changing a config value is then an SSM update and a service restart, not a full image rebuild and redeployment.

The ECS secrets block works with plain SSM String parameters, not just SecureString and Secrets Manager, and standard SSM parameters are capped at 4KB of value, so a workspace.yaml with a lot of code locations will eventually need an advanced parameter (8KB) or a different mechanism. Pipeline containers do the opposite and ship their config in the image. That’s covered in Part Two. I also changed my mind about it after writing these posts, so I add some thoughts about that as well.

Networking: the part that took me the longest

I found the networking stuff to be the most confusing. Multiple features all have to be correct before anything communicates to anything else.

What talks to what

Before any of the Terraform, it was worthwhile to me to write the traffic down explicitly, because the number of distinct flows is small and each flow has to be allowed somewhere:

graph LR
Browser(Browser)
ALB{ALB}
RW[webserver-rw]
RO[webserver-ro]
Daemon[daemon]
Uc[usercode]
Run[run task]
CM(Cloud Map)
ECSAPI(ECS API)
PG(Postgres)
S3(S3)
Browser -->|443| ALB
ALB -->|"/admin*"| RW
ALB -->|"/*"| RO
RW -->|4000| Uc
RO -->|4000| Uc
Daemon -->|4000| Uc
Daemon -->|RunTask| ECSAPI
ECSAPI -->|launches| Run
Uc -.->|register| CM
CM -.->|resolve| Daemon
CM -.->|resolve| RW
RW -->|5432| PG
Daemon -->|5432| PG
Uc -->|5432| PG
Run -->|5432| PG
Run -->|443| S3
classDef platform fill:#e8f0fe,stroke:#4285f4
classDef pipeline fill:#e6f4ea,stroke:#34a853
classDef ephemeral fill:#fef7e0,stroke:#f9ab00
classDef backend fill:#f1f3f4,stroke:#9aa0a6
class RW,RO,Daemon platform
class Uc pipeline
class Run ephemeral
class CM,ECSAPI,PG,S3 backend

Blue is the platform module, green is a pipeline, amber is the ephemeral run task, grey is something AWS runs for you. Solid arrows are request traffic, labelled with the port. Dotted arrows are service discovery, which carries no application data. Here is the same thing as a table:

From	To	Port	Why
ALB	Webserver	3000 (`webserver_port`)	Serving the UI
Webserver	Usercode server	4000	Loading code locations over gRPC
Daemon	Usercode server	4000	Polling schedules and sensors
Daemon	ECS API	443	`RunTask` to launch runs
Daemon	EC2 API	443	`DescribeNetworkInterfaces` to resolve task IPs
Daemon, Webserver, Usercode, Run	RDS Postgres	5432	Run storage, event log, schedule storage
Daemon, Webserver, Usercode, Run	S3	443	Compute logs
All tasks (at startup)	ECR, Secrets Manager, SSM, CloudWatch Logs	443	Pulling the image, injecting secrets and config

The Daemon never receives inbound traffic from anything, so its security group needs no ingress rules at all.

Three security groups

The platform module creates all three security groups, including the one used by pipelines. The usercode security group uses security group references, not CIDR blocks, to restrict inbound port 4000 to only the Daemon and Webserver, so nothing else in the VPC can reach a code server. While a CIDR rule like 10.0.0.0/16 would work too, doing so means anything that gets a private IP in that range can open a gRPC connection to your code servers, and you don’t want that. A code server could tell any caller what jobs exist and let them be launched. Referencing the daemon and webserver security groups instead means the rule stays correct as subnets change and there’s no CIDR math to get wrong.

# dagster-platform/vpc.tf

# Daemon: outbound-only. It talks to ECS APIs and usercode servers, never receives inbound.
resource "aws_security_group" "daemon" {
 name = "${var.name_prefix}-daemon-sg-${var.env}"
 description = "Dagster Daemon. Outbound only."
 vpc_id = var.vpc_id

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# Webserver: inbound from ALB only, on the webserver port.
resource "aws_security_group" "webserver" {
 name = "${var.name_prefix}-webserver-sg-${var.env}"
 description = "Dagster webservers (RO+RW). Inbound from ALB only."
 vpc_id = var.vpc_id

 ingress {
 description = "Inbound from ALB security group"
 from_port = var.webserver_port
 to_port = var.webserver_port
 protocol = "tcp"
 security_groups = [var.alb_sg_id] # reference your existing ALB SG here
 }

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# Usercode servers: inbound TCP 4000 from daemon and webserver SGs only.
# This SG is shared across all pipeline usercode services.
resource "aws_security_group" "usercode" {
 name = "${var.name_prefix}-usercode-sg-${var.env}"
 description = "Dagster usercode. Allow TCP 4000 from daemon and webserver only."
 vpc_id = var.vpc_id

 ingress {
 description = "gRPC from daemon and webserver"
 from_port = 4000
 to_port = 4000
 protocol = "tcp"
 security_groups = [
 aws_security_group.daemon.id,
 aws_security_group.webserver.id,
 ]
 }

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

The usercode security group is defined here and looked up by name in each pipeline module, so the platform owns the definition and pipelines just join it.

How do tasks get their networking?

The Dagster documentation unfortunately does not describe this clearly. There is no security group anywhere in the Terraform for run tasks. So how do they run? As I mentioned above, the answer is use_current_ecs_task_config: true in dagster.yaml. When the EcsRunLauncher launches a run, it reads the network configuration of the task making the request (the usercode code server) and reuses it. Run tasks land in the same subnets and the same usercode security group as the code server that spawned them automatically. Meaning:

A run task inherits inbound 4000 from the daemon and webserver
A run task needs the same egress as the code server: Postgres to write events, S3 for compute logs, and everything in the startup list below. Egress is where run tasks actually fail, never ingress.
If you lock the usercode security group’s egress down, you’re locking down runs too, and the failure will look like a job that starts and then hangs rather than a network error.

Each pipeline’s runs get that pipeline’s networking and that pipeline’s IAM roles, with nothing shared and nothing restated. The pipeline’s tasks are inheriting the networking rules defined for that whole pipeline.

Private Subnets Need a Path to AWS APIs

All of these tasks run in private subnets with assign_public_ip = false. A Fargate task in a private subnet still has to reach several AWS endpoints over the AWS API surface before that container code runs at all: ECR to pull the image (both the API and the layer storage in S3), Secrets Manager and SSM to inject secrets and config, and CloudWatch Logs to write those outputs. That traffic needs either a NAT gateway in the route table or VPC interface endpoints for each of those services, plus an S3 gateway endpoint. Our VPC already had a NAT gateway, so I leveraged that. If that path is missing, the task doesn’t fail in a way that mentions networking. It sits in PROVISIONING or PENDING, then stops with a ResourceInitializationError about being unable to pull the image or retrieve a secret. Nothing in the container logs, because the container never started, and nothing in the Dagster UI, because Dagster never got far enough to know about it.

What Each Failure Actually Looks Like

This table contains rough debugging notes I wish I’d had:

Symptom	Usual cause
Task stuck in `PROVISIONING`, then `ResourceInitializationError`	No egress path from the private subnet. Missing NAT route or VPC endpoints
Code location shows as failed in the UI, gRPC deadline exceeded	Usercode SG isn’t allowing 4000 from the webserver SG
Schedules and sensors never tick, but the UI loads fine	Same rule, but the daemon SG. The webserver and daemon are separate sources
Code location resolves intermittently after a deploy	Cloud Map DNS TTL still serving the old task’s IP. See part two
ALB target never goes healthy	Health check path missing the URL prefix, or the matcher rejecting a redirect
Runs launch and immediately fail with no logs	IAM, not networking. Usually the missing `ecs:TagResource`
Run starts, then hangs without writing events	Egress from the usercode SG. Run tasks inherit it

Generally, if you’re seeing an error, I’d start with examining the IAM roles first.

Service Discovery: The Namespace

The Daemon needs to find each pipeline’s code server. On ECS Fargate, the standard approach is AWS Cloud Map (private DNS). The platform creates the namespace, and each pipeline creates its own service record within it.

# dagster-platform/cloud_map.tf

resource "aws_service_discovery_private_dns_namespace" "dagster" {
 name = "pipelines-${var.env}.usercode" # e.g. "pipelines-prod.usercode"
 description = "Private DNS namespace for Dagster usercode servers"
 vpc = var.vpc_id
}

Two requirements come with a private DNS namespace. It’s bound to exactly one VPC, so a multi-VPC setup needs a namespace per VPC (which is part of why the name carries the environment). And the VPC needs both enableDnsSupport and enableDnsHostnames turned on, or the records resolve from nothing and every code location fails to load with what looks like a connection error rather than a DNS one.

IAM: Probably the crux of everything

There are two IAM roles per service: an execution role (used by ECS to pull images, write logs, and fetch secrets before the container starts) and a task role (used by the running container to call AWS APIs at runtime). Regarding AWS secrets: if the value is injected by the ECS secrets block in the task definition, the execution role needs to read it, because something needs to inject it into the task definition. If your application code calls boto3 to fetch it, the task role needs to read it. Yes, I was confused initially too.

The Shared Trust Policy

All ECS task roles share the same trust policy:

data "aws_iam_policy_document" "ecs_task_assume_role" {
 statement {
 actions = ["sts:AssumeRole"]
 principals {
 type = "Service"
 identifiers = ["ecs-tasks.amazonaws.com"]
 }
 }
}

Execution Roles (Webserver + Daemon)

Execution roles need the standard ECS managed policy, plus access to Secrets Manager and SSM so ECS can inject those values before the container starts:

# dagster-platform/iam.tf

resource "aws_iam_role" "daemon_execution" {
 name = "${var.name_prefix}-daemon-exec-role-${var.env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

resource "aws_iam_role_policy_attachment" "daemon_exec_attach" {
 role = aws_iam_role.daemon_execution.name
 policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Execution role needs Secrets Manager access so ECS can inject secrets into the container
resource "aws_iam_role_policy" "daemon_exec_db_secret" {
 name = "${var.name_prefix}-daemon-exec-db-secret-${var.env}"
 role = aws_iam_role.daemon_execution.id
 policy = data.aws_iam_policy_document.dagster_db_secret_read.json
}

# Execution role needs SSM access to inject dagster.yaml and workspace.yaml as env vars
resource "aws_iam_role_policy" "daemon_exec_ssm" {
 name = "${var.name_prefix}-daemon-exec-ssm-${var.env}"
 role = aws_iam_role.daemon_execution.id
 policy = data.aws_iam_policy_document.dagster_ssm_read.json
}

The webserver execution role is identical in structure.

Daemon Task Role

The Daemon’s task role needs to launch, describe, stop, and tag ECS run tasks. Dagster tags every run task it spawns with run metadata, and if the permission is absent, runs will launch and then immediately fail.

resource "aws_iam_role" "daemon_task" {
 name = "${var.name_prefix}-daemon-task-role-${var.env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

# Allow launching run tasks for any task def matching "pipeline-*-run-*"
# Scoped to the specific cluster so the daemon can't escape to other clusters
data "aws_iam_policy_document" "ecs_run_tasks" {
 statement {
 sid = "RunDescribeStopRunTasks"
 effect = "Allow"
 actions = ["ecs:RunTask", "ecs:DescribeTasks", "ecs:StopTask"]
 resources = [
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task-definition/pipeline-*-run-*:*",
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task/${var.cluster_name}/*",
 ]
 condition {
 test = "ArnEquals"
 variable = "ecs:cluster"
 values = [data.aws_ecs_cluster.cluster.arn]
 }
 }

 # DescribeTaskDefinition has no resource-level support, must be "*"
 statement {
 sid = "DescribeTaskDefinitions"
 effect = "Allow"
 actions = ["ecs:DescribeTaskDefinition"]
 resources = ["*"]
 }
}

resource "aws_iam_role_policy" "daemon_ecs_run" {
 name = "${var.name_prefix}-daemon-ecs-run-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_run_tasks.json
}

# Allow tagging ECS tasks. Without this, Dagster run tasks fail silently at launch.
data "aws_iam_policy_document" "ecs_tag_resource" {
 statement {
 sid = "AllowTagEcsTasks"
 effect = "Allow"
 actions = ["ecs:TagResource"]
 resources = [
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task/${var.cluster_name}/*"
 ]
 }
}

resource "aws_iam_role_policy" "daemon_ecs_tag" {
 name = "${var.name_prefix}-daemon-ecs-tag-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_tag_resource.json
}

# The daemon also needs ec2:DescribeNetworkInterfaces to resolve Fargate task IPs
resource "aws_iam_role_policy" "daemon_describe_enis" {
 name = "${var.name_prefix}-daemon-describe-enis-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ec2_describe_enis.json
}

PassRole Contract

The last piece of the daemon’s task role is the one that ties the two modules together. To launch a run task, the daemon has to pass that task’s execution and task roles to ECS, and THOSE roles live in the pipeline module, not here. The naive version grants iam:PassRole on *, which hands the daemon the ability to pass any role in the account to ECS, but that doesn’t follow the principle of least priviledge. Instead, I scoped it with tag conditions:

data "aws_iam_policy_document" "ecs_pass_dagster_roles" {
 statement {
 sid = "PassOnlyDagsterRunRoles"
 effect = "Allow"
 actions = ["iam:PassRole"]
 resources = ["*"] # resource-level not supported for PassRole; use tag conditions instead

 condition {
 test = "StringEquals"
 variable = "iam:PassedToService"
 values = ["ecs-tasks.amazonaws.com"]
 }
 # Only allow passing roles tagged as dagster pipeline components
 condition {
 test = "StringEquals"
 variable = "iam:ResourceTag/dagster:component"
 values = ["pipeline"]
 }
 condition {
 test = "StringEquals"
 variable = "iam:ResourceTag/dagster:managed-by"
 values = ["terraform"]
 }
 }
}

resource "aws_iam_role_policy" "daemon_pass_roles" {
 name = "${var.name_prefix}-daemon-pass-run-roles-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_pass_dagster_roles.json
}

iam:PassRole doesn’t support resource-level scoping the way you’d want, so "*" plus a strict tag condition is the practical answer. That means that you can add new pipelines without ever touching the platform’s IAM policies. As long as a new pipeline’s run roles carry the right tags, the daemon can pass them.

What the Platform Expects From a Pipeline

In actuality, the tag condition is really a contract. To plug into this Dagster platform, a pipeline has to:

Register a service in the Cloud Map namespace, so the Daemon and Webserver can resolve it by DNS on port 4000.
Attach its usercode service to the platform’s usercode security group, so that traffic is actually allowed.
Tag its run roles with dagster:component and dagster:managed-by, so the Daemon is permitted to pass them to ECS.
Get its code location into workspace.yaml in SSM, so the Webserver and Daemon know it exists.

Part two covers the module that does all four, plus the two-task-definition pattern, secrets, and how deploys actually roll a new image onto ECS.

Building a Schema Registry from Scratch for a Scientific Data Platform

Tue, 24 Jun 2025 09:00:00 -0700

When I joined Just-Evotec Biologics (first as a data scientist, now as a data platform engineer), I inherited a data ecosystem that probably looked pretty familiar to what lots of others have dealth with: a dozen scientific instruments each outputting data in proprietary formats, a LIMS system with its own schema, upstream and downstream experimental systems, Excel workbooks scientists had been maintaining for years but hidden away from any production storage system, and a swathe of applications that had been developed previously but had very little in the way of data governance. We didn’t have a central schema definitions, very little in the way of contracts between producers (e.g. instrumentation) and consumers (e.g. scientists, applications, clients, more instruments), and no way to ask or answer “what’s being pulled down here?”.

I’ve been slowly building up our data platform, including implementing a formal schema registry. Here, I’ll cover some core design decisions I made and patterns that worked for me for the registry. The implementation is domain (and in our case, functional group and stakeholder) agnostic.

Why not just use an existing registry tool?

Some alternative solutions to implementing / using a schema registry include Confluent’s Schema Registry, AWS Glue, Great Expectations, etc. Confluent’s Schema Registry is excellent for Kafka-based event streams, AWS Glue has a data catalog, and Great Expectations handles data quality well (just like Pandera). But none of those really addressed all or most of my needs, or fit in with my environment: a mix of validated DataFrames, LIMS API outputs, Parquet files in S3, and a team of scientists, some of whom wrote Python but weren’t data engineers. I wanted the following:

Define schemas as Python classes with static type annotations
Automatically serialize those classes to human-readable YAML
Query schemas by metadata (tier, data lineage stage, source system)
Reconstruct a live Pandera model from a stored YAML definition at runtime

We’re primarily working with data tables (as opposed to events), so I wanted a coupling between a type agnostic representation like YAML and Pandera. However, I’ve since extended a lot of this functionality for event-based schemas. At some point, I might want to revisit Glue again so I don’t reinvent the wheel.

Core architecture

The general schema development flow is: write Python → export YAML → publish to S3 → consume via the registry client at runtime.

author a Pandera contract (contracts/domains/**/schema.py)
 → export-models-to-yaml # Python model to canonical YAML
 → upload-yaml-to-s3 # publish, through the canonical-dtype gate
 S3 (what consumers actually read)
 → REGISTRY.get_pandera_model(name) # at runtime, in a pipeline
 → a live Pandera DataFrameModel

The important property of this diagram is the direction of the arrow into S3. Python is how schemas are authored, but S3 is what consumers read. A pipeline never imports a contract class. It fetches YAML and builds a Pandera model from it at runtime, which means a schema change propagates by re-publishing to S3, with no rebuild or redeploy of any consumer. I initially didn’t build this in, and found myself needed to re-install the registry every time I authored a new contract. Not ideal.

The repository is a monorepo of three independently versioned pieces:

Piece	What it is	Who uses it
the service	Contract definitions, templates, source-system specs, the FastAPI API, and the export/publish tooling	Deployed as a Lambda. Not installed as a library.
the client	Fetch schemas from S3 and deserialize them to Pandera models, plus the canonical dtype vocabulary	Pipelines, and anything validating data against a schema (e.g. dashboards)
the data IO package	Connectors for the LIMS and for the parquet/delta lakehouse	Pipelines

I started with one package and split it once pipelines started depending on it. A pipeline that just wants to validate a DataFrame should not be installing FastAPI, jsonschema, and the whole publish toolchain to do it. The client should be thin, and the dependency direction one-directional a.k.a. the service depends on the client.

Repository layout

The code tree looks like this. Names are genericized so as to hide our proprietary namespaces and functionality, but the shape is exactly what we run:

schema-registry/
├── registry/ # the service
│ ├── contracts/
│ │ ├── templates/ # JSON Schema per schema_type
│ │ │ ├── dataframe/v1_0/schema.yaml
│ │ │ ├── event/v1_0/schema.yaml
│ │ │ └── api/v1_0/schema.yaml
│ │ ├── domains/ # the contracts themselves
│ │ │ ├── lab/
│ │ │ │ └── analytics/
│ │ │ │ └── cell_count/
│ │ │ │ ├── v1_0/
│ │ │ │ │ ├── schema.py # authored
│ │ │ │ │ └── schema.yaml # generated, committed
│ │ │ │ └── v2_0/
│ │ │ │ ├── schema.py
│ │ │ │ └── schema.yaml
│ │ │ ├── operations/
│ │ │ └── enterprise/
│ │ └── utilities/dataframe/export.py # Python model → YAML
│ ├── integrations/
│ │ └── source_specs/ # contract → physical source table
│ │ └── lab/analytics/cell_count/v1_0/spec.yaml
│ ├── api/ # FastAPI app (deployed as a Lambda)
│ │ ├── app.py
│ │ ├── auth.py
│ │ └── routers/{domains,schemas,specs}.py
│ ├── publish/sync_schemas.py # validation gates + S3 upload
│ └── scripts/ # the two CLI entry points
│
└── packages/
 ├── schema-registry-client/ # what pipelines + dashboards install
 │ └── schema_registry_client/
 │ ├── contracts/utilities/dataframe/
 │ │ ├── dtypes.py # the canonical vocabulary
 │ │ ├── mixins.py # SchemaInfo, SchemaMeta, ContractMixin (e.g. constants + fields applied to all schemas)
 │ │ ├── registration.py # the decorator + path checks
 │ │ └── base.py # base DataFrameModel configs
 │ └── io/
 │ ├── registry.py # the consumer-facing Registry
 │ ├── backends/s3.py # index-backed S3 reads
 │ └── adapters/ # YAML → live Pandera model
 └── data-io/ # source-system and lakehouse connectors

I wanted to be able to seperate the client from the registry and contracts, so I want to emphasize which side of the packages/ line each concern lives on. Everything a consumer needs, the dtype vocabulary, the identity dataclasses, the deserializer, sits in the client. Everything about producing the registry, discovery, export, validation gates, upload, and the API, sits in the service. The decorator lives in the client rather than the service, which looks odd at first, but contracts are authored against it and the deserializer needs the same SchemaInfo shape when it rebuilds a model, so it belongs on the shared side.

The generated schema.yaml sitting next to the authored schema.py and getting committed is intentional. A schema change shows up in code review as a readable diff against the previous commit of the actual YAML contract, not just as a Python class that someone would have to mentally compile.

Adding a schema, end to end

The whole process for a contributor is only five steps:

Create contracts/domains/<domain>/<subdomains>/<entity>/v1_0/schema.py and write the Pandera model, decorated with @register_dataframe_schema.
If it maps to a physical source table, add the matching spec under integrations/source_specs/.
Run the export command. It walks the tree, checks that the declared name and version match the directory path, and writes schema.yaml next to each schema.py.
Commit both files. CI re-runs the export and fails the build if anything is out of date.
Run the upload command. It validates every document against its template, runs the canonical dtype gate, uploads to S3, and rewrites the index.

poetry run export-models-to-yaml # Python models → canonical schema.yaml
poetry run upload-yaml-to-s3 # validate, publish, reindex

Consumers of the registry pick the changes up on their next run, without any intermediate rebuilds or redeploys. So the rest of my post is mostly about what happens inside steps 3 and 5.

Schema identity: the domain/subdomain/entity pattern

I was very opinionated and rigid about the structure of the registry, and not just about how to define the contracts themselves. Every schema gets a unique dotted name composed of domain, zero or more subdomains, and an entity name:

 * lab.analytics.cell_count
 * lab.upstream.bioreactor_run
 * operations.equipment.instrument
 * enterprise.finance.purchase_order

This naming scheme is enforced structurally. Schemas live at file paths that mirror their names:

registry/contracts/domains/lab/analytics/cell_count/v1_0/schema.py

The @register_dataframe_schema decorator captures the identity at class definition time:

@register_dataframe_schema(
 domain="lab",
 subdomains=["analytics"],
 entity="cell_count",
 version="1.0",
 description="LIMS cell count assay table.",
 meta=SchemaMeta(
 tier="bronze",
 alignment="source",
 source_system="lims"
 )
)
class LIMSCellCountSchema(ContractMixin, LIMSBaseSchema):
 run_row_id: str = pa.Field(coerce=True)
 sample_id: str = pa.Field()
 vcd: Optional[float] = pa.Field(nullable=True, coerce=True)
 viability: Optional[float] = pa.Field(nullable=True, coerce=True)
 run_date: Timestamp = pa.Field(nullable=True, coerce=True)

The decorator is thin and only builds a frozen SchemaInfo, derives the dotted name from its parts, and stamps both onto the class:

def _register(cls):
 info = SchemaInfo(
 domain=domain,
 subdomains=subdomains,
 entity=entity,
 version=version,
 owner=owner,
 description=description,
 )
 cls.__schema_info__ = info
 cls.__schema_meta__ = meta or SchemaMeta()
 return cls

SchemaInfo.__post_init__ does a quick n’ dirty validation by rejecting any domain, subdomain, or entity containing ., /, or -, since those would corrupt the dotted name or the path it maps to, and then joins the parts into name. The more expensive checks, that the folder version matches the declared version (v1_0 → 1.0) and that the directory path matches the dotted name, run at export time, not at import time:

if RegistryConfig.enforce_domain_name:
 confirm_schema_name_matches_module(cls=cls, name=info.name)

if RegistryConfig.enforce_path_version:
 confirm_schema_version_matches_module_version(cls=cls, version=info.version)

Enforcing contract path structure at import time makes the contract classes impossible to define anywhere else, which is a problem the first time a scientist wants to draft a schema in a notebook to see what it looks like. Doing it at export means the rule is enforced on everything that gets published, while drafting stays cheap. There’s a context manager for the notebook usecase too:

with relaxed_registry_checks():
 @register_dataframe_schema(...)
 class DraftSchema(ContractMixin, BaseDataFrameSchema):
 ...

But there is a tradeoff, such a badly-placed schema fails later than it could. In practice “later” is the export step, which runs in CI on every PR, so nothing badly-placed reaches S3 regardless. Contracts inherit from a base model that sets the validation posture in one place:

class BaseDataFrameSchema(pa.DataFrameModel):
 class Config:
 strict = 'filter'
 coerce = True

strict='filter' drops columns not in the schema rather than raising, which for source-system data is almost always what you want. The LIMS base adds drop_invalid_rows = True and the row identity columns every LIMS table carries.

Versioning: semantic, path-encoded, and strict

Schema versions follow semantic versioning (1.0, 2.1, 1.0.1) and are encoded into the directory structure as Python-safe package names:

v1_0 → 1.0
v2_1 → 2.1
v1_0_1 → 1.0.1

The conversion is bidirectional and validated in both directions. When I need to add a new version of a schema, I create a new directory alongside the old one. Both continue to exist and be served by the registry.

cell_count/
 v1_0/
 schema.py
 schema.yaml ← auto-generated
 v2_0/ ← new version; v1_0 still works
 schema.py
 schema.yaml

Version resolution supports three modes: a specific version ("1.0"), "latest" (resolved by semantic comparison at read time), and None (all versions). This last mode is useful when you need to understand the full version history of a schema. Resolving latest uses packaging.version.Version for comparison rather than string sorting, so 10.0 sorts above 9.0.

for strver in vers:
 semver = Version(strver)
 # don't allow pre-releases, or dev-releases
 if semver.pre or semver.dev:
 continue
 parsed.append((semver, strver))

So I can publish 2.0.0rc1, point a pipeline at it explicitly to test, and know that nothing pinned to latest will pick it up by accident.

Export pipeline: python → YAML

Rather than maintaining YAML files by hand, the build step automatically discovers and converts decorated Pandera models. Discovery works by walking the file tree looking for schema.py files, dynamically importing them as namespace packages (which avoid collisions between modules at the same relative path), and inspecting each imported module for classes that subclass both pa.DataFrameModel and ContractMixin and have a populated __schema_info__. The code builds a unique dotted module name from the full path:

def _path_to_module_name(filepath: Path, package_root: Path) -> str:
 rel = filepath.resolve().relative_to(package_root.resolve())
 parts = rel.with_suffix("").parts
 return '.'.join(parts)
 # → "contracts.domains.lab.analytics.cell_count.v1_0.schema"

Each import gets a unique name in sys.modules to prevent collisions (ideally we wouldn’t be naming schemas the same name, but this approach allows for it). Setting mod.__package__ to the parent of that dotted name keeps relative imports inside the schema file working.

Pandera fields are serialized into a YAML-friendly dictionary, and checks that can be represented cleanly (e.g. comparisons and membership tests) are serialized. Anything else is skipped with a warning, since they’re harder to represent. Not ideal if you do want some more complicated validation logic, but works for now – though I think Great Expectations is probably better for these sorts of checks than Pandera is.

# pa.Check.greater_than(0) → {"gt": 0}
# pa.Check.isin(["A", "B"]) → {"isin": ["A", "B"]}
# lambda df: df["x"] > df["y"] → warning, skipped

The serializer returns a list per check rather than a single dict, because some Pandera checks are compound. in_range has to expand into a ge entry and an le entry, since the YAML vocabulary has no range primitive. The resulting YAML is human-readable and versionable:

schema:
 name: lab.analytics.cell_count
 domain: lab
 entity: cell_count
 version: "1.0"
 owner: data-platform
 schema_type: dataframe
 description: LIMS cell count assay table.

meta:
 tier: bronze
 alignment: source
 source_system: LIMS

columns:
 - name: run_row_id
 dtype: str
 nullable: false
 coerce: true
 - name: viability
 dtype: float64
 nullable: true
 coerce: true
 checks:
 - ge: 0.0
 - le: 1.0

Canonical dtype vocabulary

My initial version of a Python-to-YAML-to-Python round trip wrote out whatever dtype spelling the contract author (a.k.a. me) used. Pandera and pandas and typing will happily provide float, float64, double, np.float64, typing.List[float], and Timestamp for what are actually three types. I didn’t want consuming services to have to deal with this ambiguity. The registry now defines a closed dtype vocabulary:

scalars: str, int32, int64, float32, float64, bool, datetime64[ns]
lists: List[<scalar>] (element one of int/float/str/bool)

Author-side spellings are normalized to that vocabulary in exactly one function, canonicalize_dtype, called at export time:

_SCALAR_ALIASES: Dict[str, str] = {
 "int": "int64", "integer": "int64",
 "float": "float64", "double": "float64",
 "str": "str", "string": "str",
 "bool": "bool", "boolean": "bool",
 "datetime": "datetime64[ns]",
 "timestamp": "datetime64[ns]",
 "date": "datetime64[ns]",
 # ... plus the canonical spellings mapping to themselves
}

Publishing contracts

The dtype vocabulary is only a contract if something enforces it, so publishing to S3 runs every document through a hard gate first:

def assert_canonical_dtypes(doc, source="<doc>") -> None:
 schema = (doc or {}).get("schema", {})
 if schema.get("schema_type") != "dataframe":
 return

 problems = []
 for col in (doc.get("columns") or []):
 dtype = col.get("dtype")
 if dtype is None:
 continue
 try:
 canonical = canonicalize_dtype(dtype)
 except UnknownDtypeError as e:
 problems.append(f"column {col.get('name')!r}: {e}")
 continue
 if not is_canonical(dtype):
 problems.append(
 f"column {col.get('name')!r}: dtype {dtype!r} is not canonical "
 f"(expected {canonical!r}), re-export this schema."
 )

 if problems:
 raise ValueError(f"Refusing to publish {schema.get('name')!r}:\n - " + "\n - ".join(problems))

The two rejection cases are doing different jobs. Rejecting an unknown dtype catches a type consumers can’t deserialize. Rejecting a dtype that is merely non-canonical (one that canonicalize_dtype could fix) catches something else entirely: it means this YAML was not produced by the current exporter. Someone hand-edited it, or it was generated before the normalization step existed. Silently canonicalizing it at publish time would hide that. Refusing forces a re-export, which keeps the YAML in S3 in sync with the Python that claims to define it. The check acts as sort of a staleness detector.

Schema templates: validating the schemas themselves

Separately from dtypes, each schema document is validated against a JSON Schema template selected by its schema_type:

contracts/templates/
 dataframe/v1_0/schema.yaml
 event/v1_0/schema.yaml
 api/v1_0/schema.yaml

At publish time the templates are themselves checked as valid JSON Schemas with Draft202012Validator.check_schema, compiled once, and then each contract is validated against the template matching its declared type. A schema with no schema_type, or one naming a type with no template, is a hard failure, and wont upload.

This is the layer that lets the registry hold more than DataFrames. dataframe contracts carry columns. api contracts carry request, response, and callback sections. event contracts carry their own shape. They all live in the same tree, share the same identity and versioning rules, and are all published through the same gate, but each is structurally validated against its own template.

Schema metadata: queryable semantic tags

Every schema can carry structured metadata in a SchemaMeta dataclass:

@dataclass(frozen=True)
class SchemaMeta:
 tier: Optional[Literal["bronze", "silver", "gold"]] = None
 alignment: Optional[Literal["source", "canonical", "denormalized"]] = None
 contract_type: Optional[Literal["ingress", "internal", "egress"]] = None
 source_system: Optional[Union[str, List[str]]] = None
 stage: Optional[str] = None

These fields encode the data’s position in a medallion-style architecture. Bronze/silver/gold indicate the refinement level. Alignment captures whether data is still in the shape of its source system, has been canonicalized, or has been denormalized for consumption. Contract type captures directionality in the platform: ingress for uploads and instrument files, internal for the middle of pipelines, egress for published dashboards and APIs. This metadata is queryable:

registry = Registry(s3_bucket="my-data-lakehouse")

# Find all bronze schemas sourced from the LIMS
lims_bronze = registry.find_by_meta(tier="bronze", source_system="lims")

# Find all schemas used as egress contracts
egress_schemas = registry.find_by_meta(contract_type="egress")

# Get all tiers currently in use
registry.get_tiers() # → ["bronze", "gold", "silver"]

This became invaluable for impact analysis. When a source system changed its output, I could immediately identify which schemas were alignment="source" and source_system="lims" and therefore potentially affected.

The same identity travels with the data, not just the registry. ContractMixin can flatten a contract into a string key/value header for stamping onto parquet metadata:

{
 "contract.domain": "lab",
 "contract.entity": "cell_count",
 "contract.version": "1.0",
 ...
}

So a file on disk can answer which contract and which version produced it, without anyone having to consult a pipeline log.

Runtime registry

The Registry class wraps an S3 backend and provides the consumer-facing API. It’s thin by design. Most logic lives in the export pipeline and the contract definitions themselves.

from schema_registry_client import REGISTRY

# Load a schema as a live Pandera model
Model = REGISTRY.get_pandera_model("lab.analytics.cell_count", version="latest")

# Validate a DataFrame against it
validated = Model.validate(df)

# Or get the raw YAML doc
doc = REGISTRY.get_schema("lab.analytics.cell_count", version="1.0")

get_pandera_model builds a class dynamically: each YAML column becomes a Pandera Field plus a type annotation, the check dicts become Field kwargs, and the schema’s config overrides are applied on top of the base model’s config. Because the dtype strings are guaranteed canonical, the annotation lookup is a single flat dict with seven entries and no fallbacks.

The backend reads an index CSV written at upload time rather than doing live list_objects calls:

contracts/index/schema_index.csv # name, key, version, date, schema_type
integrations/index/spec_index.csv

Listing names, resolving versions, and enumerating domains are all filters over that dataframe. It loads once and stays cached on the backend instance. This keeps latency predictable and avoids a per-request S3 API call every time something enumerates the registry. The cost is that the index is a build artifact like everything else: it’s rewritten on every publish, so it’s accurate as of the last upload and no fresher.

Alongside the contracts, the registry stores integration specs that map a contract to its physical source table: schema and table name, a field_to_source column mapping, and the created/modified columns used for date partitioning. That’s what lets a pipeline say “load this contract for this date” without knowing anything about the source system’s table layout. It’s also the seam where a source system renaming a column becomes a spec change rather than a pipeline change.

API

The registry is also served over HTTP by a FastAPI app deployed as an AWS Lambda function, with Cognito authentication applied as a global dependency and a small set of public paths for health and docs.

Method	Path	Description
`GET`	`/domains/`	List all domains
`GET`	`/schemas/`	List schema names (`?domain=`, `?version=`)
`GET`	`/schemas/{name}`	A schema document (`?version=latest`)
`GET`	`/schemas/{name}/versions`	List versions for a schema
`GET`	`/schemas/{name}/diff`	Diff two versions (`?from=&to=`)
`POST`	`/schemas/{name}/validate`	Validate a payload against a schema section

The diff endpoint answers “what actually changed between 1.0 and 2.0”, which is helpful if something starts breaking after publishing schemas. The validate endpoint lets a service check a payload against an api-type contract over HTTP, without installing the client or writing Python at all. It refuses if you point it at a dataframe schema, since there’s no meaningful section to validate against.

Mostly, though, the API exists so that the registry is browsable by people who aren’t going to pip install anything. Being able to send a scientist a URL that shows exactly what a table contains has done more for adoption than any amount of documentation. I ended up building a nice dashboard that displayed datatable-specific contracts as tables, and let’s users browse by domain, subdomain(s), entity, and version. It’s much easier to look at than a YAML file or Pandera class.

Closing toughts

A schema registry is about making implicit contracts explicit. Before building this, the “contract” between a pipeline and its consumer (e.g. dashboards, APIs) was whatever the pipeline happened to output on any given run. Afterwards, it was a versioned, machine-readable document that both sides could validate against independently. None of this is domain-specific, and these same patterns would work for any environment where you need versioned, queryable, code-first schema definitions that can be serialized and distributed. I’ve incorporated contracts for everything from scientific instruments, to finance, to ERP domains in our registry.

I underestimated how much work the enforcement would be, as opposed to the contract definitions themselves. Defining schemas as Python classes is easy, and scientists who are somewhat Python-saavy can do this easily as well. Making it impossible to publish a schema that consumers can’t read, or to end up with two spellings of the same type, or to quietly delete something a pipeline still depends on, took a lot longer, and is what makes the system robust. For our small team in a domain with complex, heterogeneous data, implementing this registry and schema enforcement has been a huge boon.

I work on data platform infrastructure at a biologics company. The patterns in this post are generalized from production code, with domain-specific details abstracted away. The core framework concepts are shareable.