ECS |

Deploying Dagster to AWS ECS, Part 2: Pipelines

Wed, 05 Nov 2025 09:00:00 -0800

Here is the second of two posts on running Dagster in ECS. The first dealt with the platform side: the Daemon, the two Webservers, the ALB and Cognito wiring, the three security groups, the Cloud Map namespace, and the IAM that lets the Daemon launch anything at all.

The first component I referred to as dagster-platform. This one is about what I’m referring to as dagster-pipeline, the module you instantiate once per pipeline. Platform infrastructure gets deployed infrequently. Pipelines get deployed several times a day by whoever happens to be working on them. The reason I separated the two is that pushing a new job should never touch a security group, an IAM policy, or a load balancer rule.

The Contract

Part One ended with the four things the platform expects from a pipeline:

Register a service in the Cloud Map namespace, so the Daemon and Webserver can resolve it by DNS on port 4000.
Attach the usercode service to the platform’s usercode security group, so that traffic is actually allowed.
Tag its run roles with dagster:component and dagster:managed-by, so the Daemon is permitted to pass them to ECS.
Get its code location into workspace.yaml in SSM, so the Webserver and Daemon know it exists.

The platform defines the security group and the pipeline looks it up by name:

# dagster-pipeline/ecs.tf
data "aws_security_group" "usercode" {
 name = "${var.usercode_sg}-${var.platform_env}"
}

Here is the whole module and everything it touches. Blue is the platform, which this module does not create and only attaches to. Green is what the pipeline module builds.

graph LR
Daemon[daemon]
Web[webserver]
UcSvc[usercode service]
RunTD(run task def)
Run[run task]
CM(Cloud Map)
ECR(ECR)
SM(Secrets Manager)
PG(Postgres)
S3(S3)
SrcDB(source DB)
Web -->|4000| UcSvc
Daemon -->|4000| UcSvc
UcSvc -.->|register| CM
CM -.->|resolve| Daemon
UcSvc -.->|names| RunTD
Daemon -->|RunTask| RunTD
RunTD -.->|instantiates| Run
ECR -.->|image| UcSvc
ECR -.->|image| Run
SM -->|exec role| UcSvc
SM -->|task role| Run
UcSvc -->|5432| PG
Run -->|5432| PG
Run -->|443| S3
Run -->|query| SrcDB
classDef platform fill:#e8f0fe,stroke:#4285f4
classDef pipeline fill:#e6f4ea,stroke:#34a853
classDef ephemeral fill:#fef7e0,stroke:#f9ab00
classDef backend fill:#f1f3f4,stroke:#9aa0a6
class Daemon,Web platform
class UcSvc,CM,RunTD pipeline
class Run ephemeral
class ECR,SM,PG,S3,SrcDB backend

The run task definition is a definition, not a running thing: the usercode container names it in an environment variable, and the daemon is what actually instantiates it. The arrows pointing into the Secrets Manager land on different roles, because the usercode service has its credentials injected by ECS before it starts while the run task reads them itself at runtime.

The Two-Task-Definition Pattern

Each pipeline module creates two task definitions:

usercode task definition: the always-on code server. This runs dagster code-server start and stays alive, serving job definitions, sensors, schedules, and so on to the Daemon. It registers with Cloud Map so the platform can discover it.

run task definition: the task that executes a job. When a run is triggered, the EcsRunLauncher spins up a new Fargate task using this definition. It’s ephemeral, meaning it starts, runs the job, and exits.

The two definitions point at the same image and differ mainly in CPU/memory and in which IAM roles they use. The run definition’s command doesn’t really matter, since the launcher overrides it with the actual run command, but I set it to the same code-server command so that starting the task by hand does something reasonable. The usercode container needs to know about the run task definition so it can tell Dagster which task to launch, which can be achieved with two environment variables.

locals {
 dagster_current_image_env = [
 { name = "DAGSTER_CURRENT_IMAGE", value = data.aws_ecr_image.usercode.image_uri }
 ]

 dagster_container_context = {
 ecs = {
 task_definition_arn = aws_ecs_task_definition.run.arn
 container_name = "run"
 }
 }

 dagster_container_context_env = [
 { name = "DAGSTER_CONTAINER_CONTEXT", value = jsonencode(local.dagster_container_context) }
 ]
}

DAGSTER_CURRENT_IMAGE is the pipeline’s ECR image URI. DAGSTER_CONTAINER_CONTEXT tells Dagster which ECS task definition and container name to use for runs. Note that container_name has to match the name field of the container in the run task definition exactly. If it doesn’t, the run task launches and Dagster can’t find the container it’s supposed to be watching. Both of these go on the usercode container only, not the run container. Building DAGSTER_CONTAINER_CONTEXT from aws_ecs_task_definition.run.arn also creates the dependency that forces Terraform to build the run task definition first.

Registering With Cloud Map

The platform created the private DNS namespace, and each pipeline creates its own service record inside of that DNS namespace:

# dagster-pipeline/cloud_map.tf

# Look up the namespace created by the platform module
data "aws_service_discovery_dns_namespace" "dagster" {
 type = "DNS_PRIVATE"
 name = "pipelines-${var.env}.usercode"
}

# Each pipeline gets its own service record in the shared namespace
resource "aws_service_discovery_service" "usercode" {
 name = "${var.pipeline_name}-pipeline-${var.env}"

 dns_config {
 namespace_id = data.aws_service_discovery_dns_namespace.dagster.id

 dns_records {
 ttl = 300
 type = "A"
 }

 routing_policy = "MULTIVALUE"
 }
}

The ECS service ties itself to that record with a service_registries block, and from then on ECS registers the Fargate task’s private IP with Cloud Map automatically:

resource "aws_ecs_service" "usercode" {
 # ...
 service_registries {
 registry_arn = aws_service_discovery_service.usercode.arn
 }
}

The DNS name becomes <pipeline-name>-pipeline-<env>.pipelines-<env>.usercode, which goes into workspace.yaml:

# workspace.yaml (stored in SSM)
load_from:
 - grpc_server:
 host: ingest-pipeline-prod.pipelines-prod.usercode
 port: 4000
 location_name: "ingest"
 - grpc_server:
 host: transforms-pipeline-prod.pipelines-prod.usercode
 port: 4000
 location_name: "transforms"

“ingest” and “transforms” are what you see in the Dagster webserver when looking at each of your pipeline deployments.

TTL and Task Replacement

The ttl = 300 in that dns_records is important here. Fargate tasks get a new private IP every time they’re replaced, and a code server is replaced on every deploy. Cloud Map updates the A record immediately, but the Daemon and Webserver are resolving that name through the VPC resolver, which respects the TTL. For a few minutes after a deploy, they can be holding the IP of a task that no longer exists, and in practice, that means that a code location goes red (a.k.a looks like a failed deployment in the UI) in the UI right after a deploy and then fixes itself a few minutes later. Decreasing the TTL to 15 or 30 seconds makes deploys settle much faster, at the cost of more DNS queries, which for a handful of code servers is probably acceptable.

routing_policy = "MULTIVALUE" returns every healthy instance registered under the name. With desired_count = 1 there’s only ever one, but it’s the right policy if you later run more than one replica of a code server. Similarly, when you add a new pipeline, you need to update workspace.yaml in SSM and restart the Daemon and Webserver services so they pick up the new entry. I’m currently running this manually, but it’s worth automating in the future (but not a deal-breaker, just not something I’d consider “complete”).

Four Roles Per Pipeline

Each pipeline module creates four IAM roles: execution and task roles for both the usercode service (always-on code server) and the run task (ephemeral, launched per job execution).

Just like I mentioned in Part One, if you’re handling AWS secrets, if the value is injected by the ECS secrets block in the task definition, the execution role needs to read it. If your application code calls boto3 to fetch it, the task role needs to read it.

The roles are structurally identical, but the tags differ. The tags are what the platform’s iam:PassRole condition checks against:

# dagster-pipeline/iam.tf

locals {
 usercode_role_tags = {
 "dagster:component" = "usercode"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
 }

 run_role_tags = {
 "dagster:component" = "run"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
 }
}

# Run execution role, used by ECS to start the ephemeral run container
resource "aws_iam_role" "run_exec" {
 name = "${var.name_prefix}-${var.pipeline_name}-run-exec-role-${var.pipeline_env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
 tags = merge(local.run_role_tags, var.tags)
}

resource "aws_iam_role_policy_attachment" "run_exec_attach" {
 role = aws_iam_role.run_exec.name
 policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Run task role, used by the running job container to call AWS APIs
resource "aws_iam_role" "run_task" {
 name = "${var.name_prefix}-${var.pipeline_name}-run-task-role-${var.pipeline_env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
 tags = merge(local.run_role_tags, var.tags)
}

The Empty Resource List Trap

IAM rejects a policy statement with an empty Resource list, so a pipeline that connects to no databases can’t just get a policy document with zero ARNs in it. The document has to not exist at all:

data "aws_iam_policy_document" "db_secrets_read" {
 # A pipeline that reads no databases must not get this policy at all.
 count = length(var.db_secret_ids) > 0 ? 1 : 0

 statement {
 sid = "ReadDatabaseSecrets"
 effect = "Allow"
 actions = ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"]
 resources = [for s in data.aws_secretsmanager_secret.db_secrets : s.arn]
 }
}

resource "aws_iam_role_policy" "run_db_secrets" {
 count = length(var.db_secret_ids) > 0 ? 1 : 0
 name = "${var.name_prefix}-${var.pipeline_name}-run-db-secrets-${var.pipeline_env}"
 role = aws_iam_role.run_task.id
 policy = data.aws_iam_policy_document.db_secrets_read[0].json
}

Secrets Management

I’m using two different categories of secrets. Dagster’s own Postgres credentials are stored as a single Secrets Manager secret with JSON keys. These get injected as individual environment variables using the JSON key syntax in the ECS task definition secrets block:

locals {
 dagster_container_secrets = [
 {
 name = "DAGSTER_POSTGRES_HOSTNAME"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:hostname::"
 },
 {
 name = "DAGSTER_POSTGRES_USER"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:username::"
 },
 {
 name = "DAGSTER_POSTGRES_PASSWORD"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:password::"
 },
 {
 name = "DAGSTER_POSTGRES_DB"
 valueFrom = "${data.aws_secretsmanager_secret.dagster_postgres.arn}:name::"
 }
 ]
}

The :key:: suffix trailing fields are the version stage and version ID. When those are empty, you get the most recent / current version. Since these are injected by ECS, the execution role is what needs GetSecretValue on the secret.

Source database credentials (for pipelines that read from other databases) are not injected at all anymore. My first version of the Terraform module wrote a .pgpass file at container startup: each secret ID came in as PGPASS_SECRET_JSON_1, PGPASS_SECRET_JSON_2 and so on, plus a PGPASS_SECRET_JSON_VARS variable listing which env vars to look at, and the entrypoint fetched each one and assembled the file. It worked, but it was messy and became really annoying to manage since I was dragging around another script for building the .pgpass file. It also meant credentials sat in the container environment and on disk for the life of the task, the entrypoint had to grow a chunk of logic that had nothing to do with running jobs, so I replaced it with the following

variable "db_secret_ids" {
 type = list(string)
 description = "Secrets Manager secrets for the databases this pipeline connects to. The task role is granted GetSecretValue on them; the application reads them at runtime."
 default = []
}

import json
import boto3


def get_connection_details(secret_id: str) -> dict:
 client = boto3.client("secretsmanager")
 secret = client.get_secret_value(SecretId=secret_id)
 return json.loads(secret["SecretString"])

The credentials are never in the environment and never on disk in the task, and adding a database is one more entry in a list variable.

What I’d Do (and still might do…) Differently

Automate workspace.yaml updates. Right now, adding a pipeline requires a manual SSM update and service restart. A Lambda triggered by ECS task state changes, or even a simple CI step, could handle this.

Use Cognito + proper OIDC federation for access control. The dual-webserver approach works, but it’s not pretty. If you control your IdP, set up Cognito federation from the start. I tried setting up a proxy server to redirect traffic based on users belonging to one Cognito user-group or another, but this ended up being a huge headache.

Consider ECS Exec for debugging. I didn’t enable it initially and spent a lot of time reading logs trying to diagnose container startup issues. aws ecs execute-command is worth the extra IAM policy on the task role.

Don’t put dagster.yaml in the platform images. Storing it in SSM and injecting at runtime was the right call for the daemon and webserver. Rebuilding (EVERY) image every time you want to change a config is not the move. For pipeline images, where the code changes anyway, baking it in is fine and simpler.

Fetch application secrets at runtime instead of injecting them. See above. This is the change I’m happiest about.

Actually, One More Thing

As I was writing up this post, I went back to the IAM code to check that I’d described the tag conditions correctly, and found a bug.

Here’s the setup again. The platform’s iam:PassRole policy only allows the Daemon to pass roles tagged dagster:component = "pipeline". The pipeline module’s local says something else:

run_role_tags = {
 "dagster:component" = "run" # not "pipeline"
 "dagster:pipeline" = var.pipeline_name
 "dagster:managed-by" = "terraform"
}

"run", not "pipeline". By that reading the Daemon should never be able to pass these roles, and every run should fail. But runs work fine in both environments, so I had never noticed. The correct approach is in the IaC merge command

tags = merge(local.run_role_tags, var.tags)

merge lets later arguments win, so var.tags overrides the module’s own local. And every caller passes a tag block that happens to include exactly the key in question:

tags = {
 "env" = var.pipeline_env
 "owner" = "data-platform"
 "application" = "dagster"
 "dagster:pipeline" = var.pipeline_name
 "dagster:component" = "pipeline" # this is what makes PassRole work
 "dagster:managed-by" = "terraform"
}

So the roles do come out tagged with pipeline, and the security control does work, but only because of the argument order in a merge and a tag the caller happens to set. A new pipeline that passes a tags map without dagster:component would plan clean, apply clean, start its code server, show up in the UI, and then fail the first time someone launched a run, with an AccessDenied on PassRole and nothing in the Terraform to suggest why.

The fix is to stop letting callers override the tag the policy depends on:

tags = merge(var.tags, local.run_role_tags)

and change the platform condition to match `"run"`, which is the value the module actually controls.

Wrapping Up

Getting Dagster onto ECS was genuinely hard for me. The documentation covers the “in a perfect world” scenario, but not the real-world complexity of integrating with existing cloud infrastructure that you don’t partially (or entirely) control. There are plenty of “in a perfect world with a clean slate, here’s how you do it” walkthroughs, but very little in the way of justification or documentation of architecture choices. The platform/pipeline split in Terraform has been a huge win for me. I’ve templated our Dagster pipeline development platform and put together templated Dockerfiles and docker-compose.yaml files. Deploying a new pipeline is now a terraform apply with a handful of variables, no platform infrastructure touched, no service restarts required (except for the workspace.yaml update, which I’m working on).

Deploying Dagster to AWS ECS, Part 1: The Platform

Tue, 04 Nov 2025 09:00:00 -0800

Here at Just-Evotec Biologics, the Data Platform team uses Dagster as our orchestration platform of choice. Many of our services are deployed to AWS ECS as Fargate or EC2 tasks. Getting Dagster onto ECS took me an embarrassing number of hours, and most of those issues stemmed from a lack of documentation around getting Dagster pipelines up and running in the cloud while NOT using Dagster+ (their premium service offering). The official docs cover the happy, perfect world path, but beyond that, you’re left stitching together forum threads, GitHub issues, and a lot of trial and error. I figured I’d put together some notes on what I built, and how I built it.

The components are of this system are two-fold – the actual Python-based pipeline code, and the Terraform code I used to deploy that code to ECS. The Terraform part also comes in two pieces, because I’ve split the deployment down the middle. In this post, I’ll detail the platform half: the long-lived infrastructure everything else plugs into, such as the webserver and the daemon. Part two covers the pipeline module, what you’d instantiate per pipeline and then redeploy repeatedly as you make updates to the pipeline code.

If you came here because something in your deployment cannot reach something else, it might be helpful for you to skip straight to the networking section, covering security groups, ports, service discovery. But if not, read on…

Setup

Our team already had a multi-environment AWS setup, with VPCs, subnets, security groups, tunneling, and load balancers in place. The goal was to plug Dagster into that existing infrastructure as cleanly as possible, with proper separation of concerns. Again, as I mentioned in some other posts, I obviously can’t share our company code, so I’ll just describe some of the approaches I took.

I split the Terraform code into two distinct modules:

dagster-platform: the long-lived infrastructure. Daemon, Webserver(s), shared IAM roles, ALB listener rules, SSM parameters, Cloud Map namespace, and S3/RDS backends.
dagster-pipeline: one instantiation per pipeline. Each pipeline gets its own ECS task definitions, service, Cloud Map entry, IAM roles, and log groups. You call this module once per pipeline and it slots right into the platform.

This split makes sense from a frequency-of-interaction standpoint. The platform piece deploys infrequently, and doesn’t need to be modified much. Pipelines deploy constantly, sometimes multiple times a day, and I don’t want to touch platform infrastructure every time I push a new job.

Architecture overview

Here’s the high-level picture:

ALB (existing)
 ├── /pipelines/admin/* → Webserver (read-write) [Cognito-authenticated]
 └── /pipelines/* → Webserver (read-only) [Cognito-authenticated]

Supporting Services
 ├── RDS PostgreSQL (run storage, event log, schedule storage)
 ├── S3 (compute logs)
 ├── AWS Secrets Manager (DB credentials, application secrets)
 ├── SSM Parameter Store (dagster.yaml, workspace.yaml)
 └── Cloud Map (service discovery for usercode servers)

ECS Cluster (existing)
 ├── dagster-daemon (always-on Fargate service)
 ├── dagster-webserver-rw (always-on Fargate service)
 ├── dagster-webserver-ro (always-on Fargate service)
 └── pipeline-<name> (always-on usercode service, one per pipeline)

Everything above the pipeline-<name> line is the platform – that’s what this current post is about. The Daemon and Webserver containers share the same dagster.yaml and workspace.yaml, loaded at runtime from SSM. Pipelines register themselves via Cloud Map so the Daemon can find them.

The read-only webserver problem

Dagster’s dagster-webserver supports a --read-only flag, but there’s no built-in authentication layer that lets you say “scientists get read-only access, data engineers get admin access.” Dagster does have a subscription tier called Dagster+ that does offer this functionality, but we do not pay for that. So, we’re lefting putting our own solution in place. I’m not sure if there is a “right” answer here. One solution, given our constraints, is to combine Cognito with our IdP federation, but since I don’t control our company’s IdP tenant, I couldn’t get that working. Instead, I deployed two separate ECS services, one read-write and one read-only, each behind its own ALB target group and listener rule.

Read-write webserver command:

dagster-webserver -h 0.0.0.0 -p 3000 \
 -w /opt/dagster/dagster_home/workspace.yaml \
 -l /pipelines/admin

Read-only webserver command:

dagster-webserver -h 0.0.0.0 -p 3000 \
 -w /opt/dagster/dagster_home/workspace.yaml \
 -l /pipelines \
 --read-only

The -l flag sets the URL prefix. The ALB rules do the routing:

/pipelines/admin* → read-write target group (higher priority)
/pipelines* → read-only target group (lower priority)

Both rules are Cognito-authenticated, so anyone hitting either URL has to log in. Scientists get the read-only URL. Engineers get the admin URL. Simple, but it works.

Two actions per listener rule

An authenticated listener rule is two actions in order. The authenticate-cognito action runs first, then the forward action sends the request to the target group. If you only write the forward action, the rule works and nobody has to log in, which is the kind of mistake you don’t notice until someone tells you they never saw a login screen.

resource "aws_lb_listener_rule" "pipelines_admin_rw" {
 listener_arn = data.aws_alb_listener.webserver.arn
 priority = local.admin_listener_priority

 action {
 type = "authenticate-cognito"
 authenticate_cognito {
 user_pool_arn = data.aws_cognito_user_pool.dagster.arn
 user_pool_client_id = data.aws_cognito_user_pool_client.dagster.id
 user_pool_domain = data.aws_cognito_user_pool.dagster.domain

 on_unauthenticated_request = "authenticate"
 scope = "openid"
 session_timeout = 3600
 }
 }

 action {
 type = "forward"
 target_group_arn = aws_lb_target_group.webserver_rw.arn
 }

 condition {
 path_pattern {
 values = ["/${var.alb_rule_listener_rule_path}/admin*"]
 }
 }
}

The read-only rule is identical except for the path pattern and target group, plus a depends_on pointing at the admin rule so the two are never created in the wrong order.

Finding an open listener priority

ALB listener rule priorities have to be unique across the whole listener, and they need to be specified. Neither Terraform nor AWS will “infer” the next open value. Our ALB is shared with other applications, so hardcoding numbers means constantly checking what’s available in the console or through the CLI. I ended up writing a small bash script that the external data source calls at terraform plan time. It lists the existing rules and returns the first N unused priority slots:

data "external" "listener_priorities" {
 program = ["bash", "${path.module}/external/next_listener_priorities.sh"]

 query = {
 listener_arn = data.aws_alb_listener.webserver.arn
 region = data.aws_region.current.id
 profile = var.aws_profile
 count = 2
 }
}

locals {
 admin_listener_priority = tonumber(data.external.listener_priorities.result.priority_1)
 ro_listener_priority = tonumber(data.external.listener_priorities.result.priority_2)
}

And the script itself, which is a jq expression over describe-rules:

#!/usr/bin/env bash
set -euo pipefail

INPUT_JSON="$(cat)"
LISTENER_ARN="$(jq -r '.listener_arn' <<<"$INPUT_JSON")"
REGION="$(jq -r '.region' <<<"$INPUT_JSON")"
COUNT="$(jq -r '.count // 2' <<<"$INPUT_JSON")"

RULES_JSON="$(aws --region "$REGION" elbv2 describe-rules \
 --listener-arn "$LISTENER_ARN" --output json)"

# Find the first COUNT missing priorities in [1..50000]
jq -c --argjson count "$COUNT" '
 [ .Rules[].Priority | select(. != "default") | tonumber ] | unique as $used
 | reduce range(1; 50001) as $i
 ({out: []};
 if (.out | length) >= $count then .
 elif ($used | index($i)) == null then .out += [$i]
 else . end
 )
 | .out as $m
 | reduce range(0; ($m | length)) as $idx
 ({}; . + {("priority_" + (($idx + 1) | tostring)): ($m[$idx] | tostring)})
' <<<"$RULES_JSON"

The external data source runs on every plan, so the machine running Terraform needs the AWS CLI and jq installed and credentials that can call describe-rules (this could be your local machine, your CI/CD runner, an EC2 instance, etc.) The priorities are read at plan time, so if someone else claims a slot between your plan and your apply, the apply fails, though that’s rare enough that I’ve been willing to live with it. Basically, hardcoding priorities was not an option I wanted to entertain.

Health checks and ALB path prefixes

The ALB health checks for the webservers need to use the correct path prefix. If your webserver is mounted at /pipelines and your health check pings /, it’ll get a redirect (or a 404) and the target will never go healthy.

Read-only health check path: /<your-prefix> Read-write health check path: /<your-prefix>/admin

health_check {
 protocol = "HTTP"
 path = "/${var.alb_rule_listener_rule_path}/admin"
 matcher = "200-399"
 healthy_threshold = 2
 unhealthy_threshold = 3
 interval = 30
 timeout = 25
}

Set the matcher to 200-399 to handle redirects during startup. The Dagster webserver takes 15 to 30 seconds to come up on a cold start, so unhealthy_threshold and interval need to be generous enough that a slow container start doesn’t put the service into a restart loop.

dagster.yaml: the compute and storage specs

The dagster.yaml configuration basically let’s you define job queing, which system will run your jobs (here, ECS), where logs and compute history will be stored, etc. Here’s what our final config looks like (pretty similar to what’s already out there):

run_coordinator:
 module: dagster.core.run_coordinator
 class: QueuedRunCoordinator
 config:
 max_concurrent_runs: 15

scheduler:
 module: dagster.core.scheduler
 class: DagsterDaemonScheduler

run_launcher:
 module: dagster_aws.ecs
 class: EcsRunLauncher
 config:
 use_current_ecs_task_config: true
 include_sidecars: true
 run_task_kwargs:
 cluster: "your-ecs-cluster-name"

compute_logs:
 module: dagster_aws.s3.compute_log_manager
 class: S3ComputeLogManager
 config:
 bucket:
 env: DAGSTER_LOG_BUCKET
 prefix: "compute_logs"
 region: "your-aws-region"

run_storage:
 module: dagster_postgres.run_storage
 class: PostgresRunStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

schedule_storage:
 module: dagster_postgres.schedule_storage
 class: PostgresScheduleStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

event_log_storage:
 module: dagster_postgres.event_log
 class: PostgresEventLogStorage
 config:
 postgres_db:
 hostname: { env: DAGSTER_POSTGRES_HOSTNAME }
 username: { env: DAGSTER_POSTGRES_USER }
 password: { env: DAGSTER_POSTGRES_PASSWORD }
 db_name: { env: DAGSTER_POSTGRES_DB }
 port: 5432

retention:
 schedule:
 purge_after_days: 90
 sensor:
 purge_after_days:
 skipped: 7
 failure: 30
 success: -1

The ECS task configuration was confusing to me initially. use_current_ecs_task_config: true tells the EcsRunLauncher to take the network configuration (subnets, security groups, cluster) from the task that requested the run rather than making you restate all of it in run_task_kwargs. Combined with a per-pipeline run task definition, which is covered in part two, it means each pipeline’s runs launch into the same networking as that pipeline’s code server, with that pipeline’s own IAM roles. You don’t need to define a shared task definition across pipelines, which keeps the principle of least privilege in play.

Every credential in that file is an env: reference, and nothing secret is in the file itself.

Where the platform config lives

There are two config files: dagster.yaml (instance config, above) and workspace.yaml (the list of code locations the Daemon and Webserver load over gRPC). Baking config into the Docker image means rebuilding and redeploying the image every time you tweak a config value. For the Daemon and Webserver that’s a bad idea, because those images almost never change, but the config does. So Terraform writes both files into SSM parameters:

resource "aws_ssm_parameter" "dagster_yaml" {
 name = "${var.ssm_prefix}/dagster.yaml"
 type = "String"
 value = var.dagster_yaml
}

resource "aws_ssm_parameter" "workspace_yaml" {
 name = "${var.ssm_prefix}/workspace-${var.platform_env}.yaml"
 type = "String"
 value = var.workspace_yaml
}

and the task definitions inject them as container secrets:

locals {
 dagster_config_ssm_secrets = [
 { name = "DAGSTER_YAML", valueFrom = aws_ssm_parameter.dagster_yaml.arn },
 { name = "WORKSPACE_YAML", valueFrom = aws_ssm_parameter.workspace_yaml.arn },
 ]

 common_platform_env = [
 { name = "DAGSTER_CONFIG_FROM_ENV", value = "1" },
 { name = "DAGSTER_DEPLOYMENT_ENV", value = var.platform_env },
 { name = "DAGSTER_HOME", value = var.dagster_home },
 { name = "DAGSTER_LOG_BUCKET", value = "${var.dagster_log_bucket}-${var.platform_env}" },
 ]
}

DAGSTER_CONFIG_FROM_ENV is a flag the entrypoint script checks. When it’s set, the script writes $DAGSTER_YAML and $WORKSPACE_YAML out to $DAGSTER_HOME/dagster.yaml and $DAGSTER_HOME/workspace.yaml before starting the process, and leaves whatever is in the image alone otherwise. Changing a config value is then an SSM update and a service restart, not a full image rebuild and redeployment.

The ECS secrets block works with plain SSM String parameters, not just SecureString and Secrets Manager, and standard SSM parameters are capped at 4KB of value, so a workspace.yaml with a lot of code locations will eventually need an advanced parameter (8KB) or a different mechanism. Pipeline containers do the opposite and ship their config in the image. That’s covered in Part Two. I also changed my mind about it after writing these posts, so I add some thoughts about that as well.

Networking: the part that took me the longest

I found the networking stuff to be the most confusing. Multiple features all have to be correct before anything communicates to anything else.

What talks to what

Before any of the Terraform, it was worthwhile to me to write the traffic down explicitly, because the number of distinct flows is small and each flow has to be allowed somewhere:

graph LR
Browser(Browser)
ALB{ALB}
RW[webserver-rw]
RO[webserver-ro]
Daemon[daemon]
Uc[usercode]
Run[run task]
CM(Cloud Map)
ECSAPI(ECS API)
PG(Postgres)
S3(S3)
Browser -->|443| ALB
ALB -->|"/admin*"| RW
ALB -->|"/*"| RO
RW -->|4000| Uc
RO -->|4000| Uc
Daemon -->|4000| Uc
Daemon -->|RunTask| ECSAPI
ECSAPI -->|launches| Run
Uc -.->|register| CM
CM -.->|resolve| Daemon
CM -.->|resolve| RW
RW -->|5432| PG
Daemon -->|5432| PG
Uc -->|5432| PG
Run -->|5432| PG
Run -->|443| S3
classDef platform fill:#e8f0fe,stroke:#4285f4
classDef pipeline fill:#e6f4ea,stroke:#34a853
classDef ephemeral fill:#fef7e0,stroke:#f9ab00
classDef backend fill:#f1f3f4,stroke:#9aa0a6
class RW,RO,Daemon platform
class Uc pipeline
class Run ephemeral
class CM,ECSAPI,PG,S3 backend

Blue is the platform module, green is a pipeline, amber is the ephemeral run task, grey is something AWS runs for you. Solid arrows are request traffic, labelled with the port. Dotted arrows are service discovery, which carries no application data. Here is the same thing as a table:

From	To	Port	Why
ALB	Webserver	3000 (`webserver_port`)	Serving the UI
Webserver	Usercode server	4000	Loading code locations over gRPC
Daemon	Usercode server	4000	Polling schedules and sensors
Daemon	ECS API	443	`RunTask` to launch runs
Daemon	EC2 API	443	`DescribeNetworkInterfaces` to resolve task IPs
Daemon, Webserver, Usercode, Run	RDS Postgres	5432	Run storage, event log, schedule storage
Daemon, Webserver, Usercode, Run	S3	443	Compute logs
All tasks (at startup)	ECR, Secrets Manager, SSM, CloudWatch Logs	443	Pulling the image, injecting secrets and config

The Daemon never receives inbound traffic from anything, so its security group needs no ingress rules at all.

Three security groups

The platform module creates all three security groups, including the one used by pipelines. The usercode security group uses security group references, not CIDR blocks, to restrict inbound port 4000 to only the Daemon and Webserver, so nothing else in the VPC can reach a code server. While a CIDR rule like 10.0.0.0/16 would work too, doing so means anything that gets a private IP in that range can open a gRPC connection to your code servers, and you don’t want that. A code server could tell any caller what jobs exist and let them be launched. Referencing the daemon and webserver security groups instead means the rule stays correct as subnets change and there’s no CIDR math to get wrong.

# dagster-platform/vpc.tf

# Daemon: outbound-only. It talks to ECS APIs and usercode servers, never receives inbound.
resource "aws_security_group" "daemon" {
 name = "${var.name_prefix}-daemon-sg-${var.env}"
 description = "Dagster Daemon. Outbound only."
 vpc_id = var.vpc_id

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# Webserver: inbound from ALB only, on the webserver port.
resource "aws_security_group" "webserver" {
 name = "${var.name_prefix}-webserver-sg-${var.env}"
 description = "Dagster webservers (RO+RW). Inbound from ALB only."
 vpc_id = var.vpc_id

 ingress {
 description = "Inbound from ALB security group"
 from_port = var.webserver_port
 to_port = var.webserver_port
 protocol = "tcp"
 security_groups = [var.alb_sg_id] # reference your existing ALB SG here
 }

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

# Usercode servers: inbound TCP 4000 from daemon and webserver SGs only.
# This SG is shared across all pipeline usercode services.
resource "aws_security_group" "usercode" {
 name = "${var.name_prefix}-usercode-sg-${var.env}"
 description = "Dagster usercode. Allow TCP 4000 from daemon and webserver only."
 vpc_id = var.vpc_id

 ingress {
 description = "gRPC from daemon and webserver"
 from_port = 4000
 to_port = 4000
 protocol = "tcp"
 security_groups = [
 aws_security_group.daemon.id,
 aws_security_group.webserver.id,
 ]
 }

 egress {
 description = "Allow all outbound"
 from_port = 0
 to_port = 0
 protocol = "-1"
 cidr_blocks = ["0.0.0.0/0"]
 }
}

The usercode security group is defined here and looked up by name in each pipeline module, so the platform owns the definition and pipelines just join it.

How do tasks get their networking?

The Dagster documentation unfortunately does not describe this clearly. There is no security group anywhere in the Terraform for run tasks. So how do they run? As I mentioned above, the answer is use_current_ecs_task_config: true in dagster.yaml. When the EcsRunLauncher launches a run, it reads the network configuration of the task making the request (the usercode code server) and reuses it. Run tasks land in the same subnets and the same usercode security group as the code server that spawned them automatically. Meaning:

A run task inherits inbound 4000 from the daemon and webserver
A run task needs the same egress as the code server: Postgres to write events, S3 for compute logs, and everything in the startup list below. Egress is where run tasks actually fail, never ingress.
If you lock the usercode security group’s egress down, you’re locking down runs too, and the failure will look like a job that starts and then hangs rather than a network error.

Each pipeline’s runs get that pipeline’s networking and that pipeline’s IAM roles, with nothing shared and nothing restated. The pipeline’s tasks are inheriting the networking rules defined for that whole pipeline.

Private Subnets Need a Path to AWS APIs

All of these tasks run in private subnets with assign_public_ip = false. A Fargate task in a private subnet still has to reach several AWS endpoints over the AWS API surface before that container code runs at all: ECR to pull the image (both the API and the layer storage in S3), Secrets Manager and SSM to inject secrets and config, and CloudWatch Logs to write those outputs. That traffic needs either a NAT gateway in the route table or VPC interface endpoints for each of those services, plus an S3 gateway endpoint. Our VPC already had a NAT gateway, so I leveraged that. If that path is missing, the task doesn’t fail in a way that mentions networking. It sits in PROVISIONING or PENDING, then stops with a ResourceInitializationError about being unable to pull the image or retrieve a secret. Nothing in the container logs, because the container never started, and nothing in the Dagster UI, because Dagster never got far enough to know about it.

What Each Failure Actually Looks Like

This table contains rough debugging notes I wish I’d had:

Symptom	Usual cause
Task stuck in `PROVISIONING`, then `ResourceInitializationError`	No egress path from the private subnet. Missing NAT route or VPC endpoints
Code location shows as failed in the UI, gRPC deadline exceeded	Usercode SG isn’t allowing 4000 from the webserver SG
Schedules and sensors never tick, but the UI loads fine	Same rule, but the daemon SG. The webserver and daemon are separate sources
Code location resolves intermittently after a deploy	Cloud Map DNS TTL still serving the old task’s IP. See part two
ALB target never goes healthy	Health check path missing the URL prefix, or the matcher rejecting a redirect
Runs launch and immediately fail with no logs	IAM, not networking. Usually the missing `ecs:TagResource`
Run starts, then hangs without writing events	Egress from the usercode SG. Run tasks inherit it

Generally, if you’re seeing an error, I’d start with examining the IAM roles first.

Service Discovery: The Namespace

The Daemon needs to find each pipeline’s code server. On ECS Fargate, the standard approach is AWS Cloud Map (private DNS). The platform creates the namespace, and each pipeline creates its own service record within it.

# dagster-platform/cloud_map.tf

resource "aws_service_discovery_private_dns_namespace" "dagster" {
 name = "pipelines-${var.env}.usercode" # e.g. "pipelines-prod.usercode"
 description = "Private DNS namespace for Dagster usercode servers"
 vpc = var.vpc_id
}

Two requirements come with a private DNS namespace. It’s bound to exactly one VPC, so a multi-VPC setup needs a namespace per VPC (which is part of why the name carries the environment). And the VPC needs both enableDnsSupport and enableDnsHostnames turned on, or the records resolve from nothing and every code location fails to load with what looks like a connection error rather than a DNS one.

IAM: Probably the crux of everything

There are two IAM roles per service: an execution role (used by ECS to pull images, write logs, and fetch secrets before the container starts) and a task role (used by the running container to call AWS APIs at runtime). Regarding AWS secrets: if the value is injected by the ECS secrets block in the task definition, the execution role needs to read it, because something needs to inject it into the task definition. If your application code calls boto3 to fetch it, the task role needs to read it. Yes, I was confused initially too.

The Shared Trust Policy

All ECS task roles share the same trust policy:

data "aws_iam_policy_document" "ecs_task_assume_role" {
 statement {
 actions = ["sts:AssumeRole"]
 principals {
 type = "Service"
 identifiers = ["ecs-tasks.amazonaws.com"]
 }
 }
}

Execution Roles (Webserver + Daemon)

Execution roles need the standard ECS managed policy, plus access to Secrets Manager and SSM so ECS can inject those values before the container starts:

# dagster-platform/iam.tf

resource "aws_iam_role" "daemon_execution" {
 name = "${var.name_prefix}-daemon-exec-role-${var.env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

resource "aws_iam_role_policy_attachment" "daemon_exec_attach" {
 role = aws_iam_role.daemon_execution.name
 policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

# Execution role needs Secrets Manager access so ECS can inject secrets into the container
resource "aws_iam_role_policy" "daemon_exec_db_secret" {
 name = "${var.name_prefix}-daemon-exec-db-secret-${var.env}"
 role = aws_iam_role.daemon_execution.id
 policy = data.aws_iam_policy_document.dagster_db_secret_read.json
}

# Execution role needs SSM access to inject dagster.yaml and workspace.yaml as env vars
resource "aws_iam_role_policy" "daemon_exec_ssm" {
 name = "${var.name_prefix}-daemon-exec-ssm-${var.env}"
 role = aws_iam_role.daemon_execution.id
 policy = data.aws_iam_policy_document.dagster_ssm_read.json
}

The webserver execution role is identical in structure.

Daemon Task Role

The Daemon’s task role needs to launch, describe, stop, and tag ECS run tasks. Dagster tags every run task it spawns with run metadata, and if the permission is absent, runs will launch and then immediately fail.

resource "aws_iam_role" "daemon_task" {
 name = "${var.name_prefix}-daemon-task-role-${var.env}"
 assume_role_policy = data.aws_iam_policy_document.ecs_task_assume_role.json
}

# Allow launching run tasks for any task def matching "pipeline-*-run-*"
# Scoped to the specific cluster so the daemon can't escape to other clusters
data "aws_iam_policy_document" "ecs_run_tasks" {
 statement {
 sid = "RunDescribeStopRunTasks"
 effect = "Allow"
 actions = ["ecs:RunTask", "ecs:DescribeTasks", "ecs:StopTask"]
 resources = [
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task-definition/pipeline-*-run-*:*",
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task/${var.cluster_name}/*",
 ]
 condition {
 test = "ArnEquals"
 variable = "ecs:cluster"
 values = [data.aws_ecs_cluster.cluster.arn]
 }
 }

 # DescribeTaskDefinition has no resource-level support, must be "*"
 statement {
 sid = "DescribeTaskDefinitions"
 effect = "Allow"
 actions = ["ecs:DescribeTaskDefinition"]
 resources = ["*"]
 }
}

resource "aws_iam_role_policy" "daemon_ecs_run" {
 name = "${var.name_prefix}-daemon-ecs-run-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_run_tasks.json
}

# Allow tagging ECS tasks. Without this, Dagster run tasks fail silently at launch.
data "aws_iam_policy_document" "ecs_tag_resource" {
 statement {
 sid = "AllowTagEcsTasks"
 effect = "Allow"
 actions = ["ecs:TagResource"]
 resources = [
 "arn:aws:ecs:${var.region}:${data.aws_caller_identity.current.account_id}:task/${var.cluster_name}/*"
 ]
 }
}

resource "aws_iam_role_policy" "daemon_ecs_tag" {
 name = "${var.name_prefix}-daemon-ecs-tag-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_tag_resource.json
}

# The daemon also needs ec2:DescribeNetworkInterfaces to resolve Fargate task IPs
resource "aws_iam_role_policy" "daemon_describe_enis" {
 name = "${var.name_prefix}-daemon-describe-enis-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ec2_describe_enis.json
}

PassRole Contract

The last piece of the daemon’s task role is the one that ties the two modules together. To launch a run task, the daemon has to pass that task’s execution and task roles to ECS, and THOSE roles live in the pipeline module, not here. The naive version grants iam:PassRole on *, which hands the daemon the ability to pass any role in the account to ECS, but that doesn’t follow the principle of least priviledge. Instead, I scoped it with tag conditions:

data "aws_iam_policy_document" "ecs_pass_dagster_roles" {
 statement {
 sid = "PassOnlyDagsterRunRoles"
 effect = "Allow"
 actions = ["iam:PassRole"]
 resources = ["*"] # resource-level not supported for PassRole; use tag conditions instead

 condition {
 test = "StringEquals"
 variable = "iam:PassedToService"
 values = ["ecs-tasks.amazonaws.com"]
 }
 # Only allow passing roles tagged as dagster pipeline components
 condition {
 test = "StringEquals"
 variable = "iam:ResourceTag/dagster:component"
 values = ["pipeline"]
 }
 condition {
 test = "StringEquals"
 variable = "iam:ResourceTag/dagster:managed-by"
 values = ["terraform"]
 }
 }
}

resource "aws_iam_role_policy" "daemon_pass_roles" {
 name = "${var.name_prefix}-daemon-pass-run-roles-${var.env}"
 role = aws_iam_role.daemon_task.id
 policy = data.aws_iam_policy_document.ecs_pass_dagster_roles.json
}

iam:PassRole doesn’t support resource-level scoping the way you’d want, so "*" plus a strict tag condition is the practical answer. That means that you can add new pipelines without ever touching the platform’s IAM policies. As long as a new pipeline’s run roles carry the right tags, the daemon can pass them.

What the Platform Expects From a Pipeline

In actuality, the tag condition is really a contract. To plug into this Dagster platform, a pipeline has to:

Register a service in the Cloud Map namespace, so the Daemon and Webserver can resolve it by DNS on port 4000.
Attach its usercode service to the platform’s usercode security group, so that traffic is actually allowed.
Tag its run roles with dagster:component and dagster:managed-by, so the Daemon is permitted to pass them to ECS.
Get its code location into workspace.yaml in SSM, so the Webserver and Daemon know it exists.

Part two covers the module that does all four, plus the two-task-definition pattern, secrets, and how deploys actually roll a new image onto ECS.