Data Platform

Correlation IDs and Request Lineage Across Services

When a scientist says a dashboard was slow this morning, our logs couldn't tell us which requests were theirs. I built a request context, stored in a ContextVar and forwarded as HTTP headers, that stamps request, session, user, and service IDs onto every log line. The post covers the middleware, why a logging filter beats a LoggerAdapter, and fixes for noisy health checks and unhelpful 403s.

Service-to-Service Auth with Cognito Scopes

Why we replaced a shared bearer token with Cognito's client-credentials flow, so each service has its own identity and each endpoint requires a specific scope. Covers resource servers, custom scopes, token validation, and a couple of ALB rule gotchas.

Deploying Dagster to AWS ECS, Part 2: Pipelines

The per-pipeline Terraform module that plugs into the platform, with two task definitions per pipeline, Cloud Map registration so the daemon can find each code server, four IAM roles, and secrets handling.

Deploying Dagster to AWS ECS, Part 1: The Platform

The long-lived half of our Dagster deployment on AWS ECS: the daemon, read-only and admin webservers behind a Cognito-authenticated ALB, and the Terraform that ties them together. Most of the post is about networking, security groups, service discovery, and IAM, which is where most of my time went. Written for anyone running Dagster without Dagster+.