Middleware |

Correlation IDs and Request Lineage Across Services

Tue, 27 Jan 2026 09:00:00 -0800

We are a pretty small team, yet we still have a pretty large and expansive suite of services, dashboards, and backend data systems. When the number of services was small, auditting and logging was quite easy, but over the last few years, it’s become quite a chore. Imagine the following scenario:

A scientist tells you the page on the cell culture analysis dashboard was slow this morning.

If you can’t answer at least one of the following, you might find this post helpful:

Which page?
What do they mean “slow”?
Which user?
When?
What endpoint and what parameters?

The dashboard they’re referring to called some backend API, that API called two more, and one of those queried the LIMS system directly. That’s multiple services and log streams, and I’m willing to bet that not one line in any of them tells you which entries belong to that user’s click.

We face this issue in a variety of places in our data platform. I don’t mean logging in the general sense of import logging, but the narrower scope of how a request identifies itself as it moves between all of our services, so that afterwards we can put the pieces back in order and say which step was slow, or which one returned the 403 error.

I built something that I’ve been slowly deploying and incorporating into most of our services and dashboards. Three pieces do the work: a request-scoped context, middleware that fills it in, and a handful of headers forwarded on every outbound call. It pairs with the previous post on service-to-service auth, since knowing who called and when is half the battle.

What I wanted every line to carry

My goal for this tooling was that any single “typical” log line, anywhere in the system, can answer which request, which user, which service, which endpoint, how long, and what happened. This ends up being a fixed set of fields stamped onto every record key:

Field	Example	Why
`request_id`	`req-474c2493-...`	Ties every line from one request together, across services
`session_id`	`ses-a55813da-...`	Ties multiple requests from one browser session together
`user_id`	`user@example.com`	Who was actually doing this
`service`	`lims-adapter`	Which service emitted the line
`requesting_service`	`dashboard`	Which service called this one
`method` / `path`	`GET` `/samples/S-123/molecule`	Which endpoint
`client`	`10.0.5.233`	Source IP
`status_code`	`403`	What happened
`duration_s`	`1.284`	How long it took
`event`	`request_end`	Machine-readable label for the kind of line

From my perspective, request_id and session_id are the most helpful. A request ID covers one call to a service and everything it fans out into. A session ID covers everything one person did in one sitting. When someone says “it was slow this morning”, you find their session, then look at which requests inside it were slow.

How the Context Travels

There are two mechanisms for how context travels within and between requests, one inside a process and one between processes.

graph LR
Browser(Browser)
Dash[Dashboard]
ApiA[Service A]
ApiB[Service B]
Lims[LIMS]
Logs[Log store]
Browser -->|"click"| Dash
Dash -->|"X-Request-ID, X-Session-ID, X-User-ID, X-Requesting-Service: dashboard"| ApiA
ApiA -->|"same IDs, X-Requesting-Service: service-a"| ApiB
ApiA -->|"same IDs"| Lims
Dash -.->|"req-abc"| Logs
ApiA -.->|"req-abc"| Logs
ApiB -.->|"req-abc"| Logs
classDef store fill:#f3e8fd,stroke:#a142f4
class Logs store

Inside a process, the fields live in a ContextVar, so they follow the request through await chains and across threads without being passed as arguments. Between processes, they’re plain HTTP headers that outbound calls forward. X-Requesting-Service is the one that changes at each hop, while the other three carry through unchanged. That is what lets one request_id span the entire tree.

Why use a filter, not a LoggerAdapter

One way to attach context to log records is by using Python’s logging.LoggerAdapter, but I think it’s the wrong tool for what I’m trying to do. An adapter applies only to log calls made through it. Your own code gets the context, but the third-party library that raises the interesting exception does not, because it holds its own plain logging.getLogger(__name__).

A logging.Filter attached to the root handler sees every record in the process, irrespective of where it came from:

_log_context: ContextVar[dict] = ContextVar("log_context", default={})


class ContextFilter(logging.Filter):
 """Inject current context fields onto every LogRecord."""

 def filter(self, record: logging.LogRecord) -> bool:
 for key, value in {**_global_fields, **_log_context.get()}.items():
 if not hasattr(record, key):
 setattr(record, key, value)
 return True

The hasattr check means a field passed explicitly at the call site wins over the passive default context, so a caller can override user_id for one line without unbinding it. And _global_fields is a plain dict rather than a ContextVar, because ContextVar values aren’t inherited by new threads: static configuration like the service name has to live outside the context or it vanishes in a thread pool.

Binding is a context manager that nests:

@contextmanager
def log_context(**fields):
 current = _log_context.get()
 token = _log_context.set({**current, **fields})
 try:
 yield
 finally:
 _log_context.reset(token)

with log_context(request_id=rid, session_id=sid):
 log.info("request received") # two fields
 with log_context(step="validate"):
 log.info("validating") # three
 log.info("continuing") # back to two

The token mechanism restores the previous context exactly, including when the block exits by exception, which a naive save-and-restore gets wrong.

There’s also a version without a scope boundary, for values that resolve partway through a request. Identity is the usual case: you don’t know the user until auth has run, but you want every line after that point to carry it.

async def get_current_user(token: str) -> User:
 user = decode_token(token)
 update_log_context(user_id=user.email) # every later line carries it
 return user

Middleware

The middleware operates once per request: accept or generate the IDs, resolve the user, bind everything, time the request, and echo the IDs back.

request_id = request.headers.get("x-request-id") or f"req-{uuid.uuid4()}"
session_id = request.headers.get("x-session-id") or f"ses-{uuid.uuid4()}"

Accept-or-generate is what enables cross-service correlation. The first service to see a request mints the ID, and every service after that inherits the minted ID from the header. Echoing the request and session IDs back in the response headers means a client, a browser console, or a support ticket can name the exact request, which turns “it was slow this morning” into a string I can now search for in CloudWatch. Identity resolves from whichever source is available, in priority order:

x-amzn-oidc-data, the ALB-injected Cognito JWT, decoded for the email claim. This is a direct browser hit.
X-User-ID, forwarded by an upstream service that already decoded its own JWT.
A dev stub, when APP_ENV is dev, local, or unset.

The second tier carries a user’s identity into services that the user never talks to directly. Service B has no OIDC header, because its caller was service A, not a browser. It can still log which person’s click ultimately caused the call.

Grading by status

request_end is logged at ERROR for 5xx, WARNING for 4xx, and INFO otherwise:

log.log(
 logging.ERROR if status >= 500
 else logging.WARNING if status >= 400
 else logging.INFO,
 "request completed",
 extra={"event": "request_end", "status_code": status, "duration_s": duration_s},
)

403s that explain nothing

A route raises HTTPException(403, detail="missing scope: service-b/write"). The client gets a useful error. The log gets a bare 403 and nothing else. The reason is ordering: FastAPI’s ExceptionMiddleware sits inside the middleware, so by the time the middleware sees anything, the exception has already been converted into a response. The detail explaining the rejection went to the client and nowhere else. At that point, debugging a 403 means reproducing it, which is annoying and what this whole codebase is meant to fix.

The fix is an exception handler that records the reason, registered alongside the middleware:

async def http_exception_log_handler(request, exc):
 log.log(
 logging.ERROR if exc.status_code >= 500 else logging.WARNING,
 "request rejected",
 extra={
 "event": "request_rejected",
 "status_code": exc.status_code,
 "error_detail": _detail_text(exc.detail),
 },
 )
 return await _default_http_exception_handler(request, exc)

app.add_exception_handler(StarletteHTTPException, http_exception_log_handler)

It runs inside the middleware’s context, so the request, session, and user fields are already bound. It delegates to Starlette’s default handler, so the response the client receives is unchanged. And because detail is often a dict for structured errors, it gets JSON-serialized rather than str()-ed, so it stays greppable instead of turning into Python repr with single quotes.

Health probe overload

A load balancer probes /health every few seconds from every node, which means that those log lines can dominate log volume while reporting only that nothing happened, especially if they’re occuring frequently. I chose to suppress them, while still retaining useful information in the case that they fail:

Skip the request_start/request_end pair in the middleware for passing probes.
Drop uvicorn’s own access-log line for the same requests, via a filter on the root handler.

Uvicorn logs independently of your middleware, so leaving that half in place means you’ve halved the noise and kept the volume. A probe returning 4xx or 5xx is always logged – that’s the whole point of building this logging functionality. The correlation headers are still echoed on suppressed requests too, so if a probe starts failing and something retries, the retry is traceable.

Auto-capturing tracebacks

A log call made from inside an except block captures the active exception automatically, even when the caller writes log.warning("call failed") with no exc_info=True. I typically write logger.warning in error handlers constantly, and almost never remember exc_info. This results in a log full of “call failed” with no stacktrace. Capturing it by default means the traceback is there whether or not the author thought about it.

The record then carries both locations, which are usually different:

{
 "timestamp": "2026-05-26T19:09:37.015695Z",
 "level": "WARNING",
 "service": "lims-adapter",
 "message": "LIMS call failed: SQL query failed (upstream_status=405)",
 "func": "_lims_error",
 "filename": "errors.py",
 "lineno": 47,
 "request_id": "req-474c2493-18ea-42a0-be0f-5ce4d096b14c",
 "session_id": "ses-a55813da-3ab1-4471-9761-59d2ef988179",
 "user_id": "user@example.com",
 "path": "/samples/S-20260129-269/molecule",
 "exception": "Traceback (most recent call last):\n File \".../molecule.py\", line 10, in get_molecule\n ...\nLimsError: SQL query failed"
}

filename and lineno point at the error-handling code: the place someone decided to log. The traceback points somewhere else entirely, at the origin of the raise. That is where the bug lives. You want both, and they are rarely in the same file.

Field names and markers

Constants instead of string literals. A typo in fields.REQUEST_ID is a NameError at import. A typo in "request_id" is a silent schema divergence that nobody notices until a dashboard query quietly returns fewer rows than it should.

from logging_lib import fields

with log_context(**{fields.REQUEST_ID: rid, fields.USER_ID: uid}):
 log.info("handling request", extra={fields.EVENT: "request_start"})

Markers for categories. A top-level marker field tags a record for dashboards and alerting: AUDIT for user-initiated state changes, PERF for timings, SECURITY for auth and access control, SYSTEM for lifecycle, PIPELINE for job execution. It’s a small closed vocabulary, which is the point, and it makes queries trivial in whatever you’re storing logs in:

# CloudWatch Logs Insights
filter marker = "AUDIT"

# Loki
{service="lims-adapter"} | json | marker = "AUDIT"

What I now get

With the IDs in place, three questions become queries rather than investigations.

Sequencing. Filter on one request_id, sort by timestamp, and I have the ordered list of everything every service did for that one call, including the services the user never touched directly.

Lineage. requesting_service on each hop gives the call graph as it actually ran

Latency attribution. Every service records duration_s for the same request_id. The dashboard reports 3.1 seconds, service A reports 2.9, service B reports 2.7, and the LIMS query reports 2.6. The slow thing is the LIMS query, and it’s established without adding a single timer.

Service-to-Service Auth with Cognito Scopes

Tue, 09 Dec 2025 09:00:00 -0800

I’m building tools that help scientists run their day-to-day work. I work in a heavily regulated environment, subject to both U.S. and European digital policy requirements. The scenario we face regularly is: two services talk to each other. Then three. A dashboard calls a backend API, that API calls another one, a nightly job calls all of them. Somewhow, we have to answer how one service proves to another that it is allowed to make the call.

The first way I addressed this was to use Internal Bearer Tokens. Basically, you generate a long random string, store it in the AWS Secrets Manager, inject it into every service as an environment variable, have each service check inbound requests for it.

I replaced that solution with Cognito’s OAuth2 client-credentials flow. Here each service carries its own identity and each endpoint declares the scope it requires. We didn’t need to define any new infrastructure for this, since Cognito was already authenticating browser users through the ALB, so the service-to-service (S2S) path ended up reusing the same user pool, the same JWKS endpoint, and the same validation code that was already there.

Why a shared bearer token was not viable

The bearer token approach was really easy, but it wasn’t a viable approach in the long run, for a few reasons:

No expiry. A static token has no TTL. If it leaks through a log line, a memory dump, or a compromised container, it stays valid until you manually rotate it and run a redeploy of every service holding it. There is no safe window during rotation where both the old and new values work, unless you write that special-casing yourself.

No identity. When service B receives the token, it can verify the value matches what it was given, but it can’t tell who sent it. Every caller is indistinguishable, so you can only observe that the token was used.

No scope. The receiving service has no way to enforce what the caller may do. Either it accepts the token and grants full access, or it rejects it. There’s no way to express “service A may read but not write”.

Shared blast radius. The same secret works everywhere. One compromised service exposes the key to every other service, so a single breach leaves a huge hole.

Comparing these two solutions against each other, we have:

Property	Static bearer token	Cognito client credentials
Expiry	Never	~1 hour, auto-refreshed
Identity	None (opaque string)	Signed JWT with a `client_id` claim
Scope enforcement	Impossible	Per-endpoint, per-caller
Compromise blast radius	All services, indefinitely	One service, for TTL
Rotation	Manual, coordinated redeploy	Automatic
Auditability	“the token was used”	“service A called endpoint X”

Being able to attribute traffic to a caller enhances the utility of logs, which is most of the subject of my next post.

Two gates

There are two authentication and authorization gates in this architecture:

Gate 1 is the ALB. Everything in our system is either accesible from outside our VPC via the ALB, or has ingress/egress restricted to within the VPC. The ALB decides only how a request gets in: either this is an authenticated browser user who must complete an interactive Cognito login, or it’s a caller to hand straight to the service. The ALB knows nothing about your application’s permissions.

Gate 2 is the service. It validates identity and enforces scope. This is the real authorization check for S2S communication. It behaves identically no matter what language or host the caller runs on.

An incoming request has to clear both. A browser user who hasn’t been authenticated gets a 302 status to the Cognito login page via Gate 1 and never reaches the dashboard or API. A service with a valid token but the wrong scope passes Gate 1 but receives a 403 at Gate 2.

caller ──▶ [ Gate 1: ALB ] ──▶ [ Gate 2: service ] ──▶ your handler
 browser or machine? who are you, and what may you do?
 │ browser, │ no or invalid credential
 │ not logged in │ valid token, wrong scope
 ▼ ▼
 302 → Cognito login 401 / 403

What services can call what

Below is a diagram of allowed communication. Solid arrows are requests, and each one is labeled with the scope it must present. Dashed arrows are token issuance.

graph LR
Human(Browser user)
Robot(Laptop / CI job)
ALB{ALB}
Dash[Dashboard]
SvcA[Service A]
SvcB[Service B]
SvcC[Service C]
Cognito(Cognito)
Human -->|"session cookie, no scope"| ALB
Robot -->|"Authorization: Bearer"| ALB
ALB -->|"x-amzn-oidc-identity"| Dash
ALB -->|"Bearer passthrough"| SvcA
Dash -->|"service-a/read"| SvcA
Dash -->|"service-b/read"| SvcB
SvcA -->|"service-b/read"| SvcB
SvcA -->|"service-c/write"| SvcC
Cognito -.->|"token: service-a/read, service-b/read"| Dash
Cognito -.->|"token: service-b/read, service-c/write"| SvcA
classDef human fill:#e8f0fe,stroke:#4285f4
classDef machine fill:#e6f4ea,stroke:#34a853
class Human human
class Robot machine

A browser user reaches the dashboard with no scope at all, because the ALB already authenticated them interactively and the dashboard is the thing they’re allowed to look at. The dashboard’s own token is narrower than service A’s: it can read service A and service B, but nothing grants it write access to service C. If the dashboard is compromised, the write path isn’t reachable from there.

The permissions are just a table, so adding a consumer means adding a row, not changing any service’s code:

App client (the caller)	Allowed scopes
`dashboard`	`service-a/read`, `service-b/read`
`service-a`	`service-b/read`, `service-c/write`
`ci-runner`	`service-a/read`
`developer`	`service-a/read`, `service-b/read`

Cognito resource servers, custom scopes, and app clients

The resource server defines capabilities

A resource server represents a service that exposes capabilities. It has two name-like fields that do very different jobs:

# terraform code

resource "aws_cognito_resource_server" "service_b" {
 user_pool_id = var.cognito_user_pool_id

 # The scope prefix. Opaque string, conventionally a URL. Immutable in practice.
 identifier = "https://api.example.com/service-b"

 # Console display name only. Nothing references it.
 name = "service-b"

 scope {
 scope_name = "read"
 scope_description = "Read access to service-b"
 }

 scope {
 scope_name = "write"
 scope_description = "Write access to service-b"
 }

 scope {
 scope_name = "admin"
 scope_description = "Administrative access to service-b"
 }
}

The identifier field becomes the prefix of every scope the resource server defines, so the three scopes above are really:

https://api.example.com/service-b/read
https://api.example.com/service-b/write
https://api.example.com/service-b/admin

That full string is what a client requests, what appears in the JWT’s scope claim, and what a service compares against. name is a display label in the console and is referenced by nothing.

Only custom scopes work for machine-to-machine

The client-credentials flow works only with custom scopes from a resource server. I defined these scope manually in my FastAPI applications, and decorated the relevant endpoints with their respective scopes. The built-in OpenID scopes (openid, email, profile, aws.cognito.signin.user.admin) are for user-facing flows, and asking for one with grant_type=client_credentials fails. Every machine caller needs at least one resource server to exist, even if the service it calls has exactly one capability.

The app client also needs a secret and the flow explicitly enabled. A client without generate_secret cannot do client credentials at all, and the user pool needs a domain configured, since the token endpoint lives at https://<domain>.auth.<region>.amazoncognito.com/oauth2/token.

The app client selects a subset

An app client represents a caller of a service. One per calling service, with allowed_oauth_scopes listing every downstream capability it may request:

resource "aws_cognito_user_pool_client" "service_a_m2m" {
 name = "service-a-m2m"
 user_pool_id = var.cognito_user_pool_id

 generate_secret = true
 allowed_oauth_flows_user_pool_client = true
 allowed_oauth_flows = ["client_credentials"]

 # Full scope strings, exactly as composed above.
 allowed_oauth_scopes = [
 "https://api.example.com/service-b/read",
 "https://api.example.com/service-c/write",
 ]

 # Cognito validates that each scope exists, so the resource servers
 # must be created first. Terraform will not infer this ordering.
 depends_on = [
 aws_cognito_resource_server.service_b,
 aws_cognito_resource_server.service_c,
 ]
}

Service A is granted read on service B, and write on service C. It cannot write to service B even though that scope exists, because its app client never lists it as a viable permissions. Cognito rejects an app client referencing a scope that doesn’t exist yet, and Terraform sees no dependency between the two resources because the scope is a hand-written string rather than a reference to the resource server’s attributes.

Scopes model capabilities, not consumers. Service B declares that reading, writing, and administering are things that can be done to it (this is defined both in the FastAPI code via decorated endpoints, and in the Cognito resource server). But Service B doesn’t say anything about who can take those actions. Consumers get assigned action permissions through their app client’s allowed-scopes list.

That separation lets you onboard a caller without touching the service being called. A new consumer is one app client and one secret. Service B does not redeploy, does not learn the new caller’s name, and does not grow a config entry. Its code already says require_scope("read") and will keep saying that no matter how many services eventually call it.

Two possible gotchas

An omitted scope parameter is not the same as no scopes. If a token request leaves scope off entirely, Cognito issues a token carrying every scope the app client is allowed. This means that every downstream call then presents maximum privilege. Always request scopes explicitly, one per downstream call. This is also why caching tokens per scope rather than per client matters.

Removing a scope is harder than adding one. Cognito will not let you delete a scope from a resource server while an app client still lists it in allowed_oauth_scopes. The order is: remove it from every app client, apply, then remove it from the resource server. In Terraform that’s two applies, and doing it in one produces an error that names the resource server rather than the app client that’s actually the problem.

Validating a token

The receiving side has to check four things:

The signature, against Cognito’s public JWKS for the pool.
The issuer, so a token from some other pool isn’t accepted.
token_use == "access", so an ID token can’t be substituted for an access token.
The scope claim contains the specific scope the endpoint requires.

def _decode_token(self, token: str) -> dict:
 signing_key = self._jwk_client.get_signing_key_from_jwt(token)

 claims = jwt.decode(
 token,
 signing_key.key,
 algorithms=["RS256"],
 issuer=self.issuer,
 # Client-credentials tokens carry `client_id` instead of `aud`,
 # so skip audience verification and check token_use instead.
 options={"verify_aud": False},
 )

 if claims.get("token_use") != "access":
 raise HTTPException(401, "wrong token type")

 return claims

Client-credentials tokens have no aud claim. Instead, they carry client_id , so leaving audience verification on rejects every valid S2S token with a confusing error. Turning it off here is valid, but only because token_use and the issuer are being checked instead.

Scope enforcement then becomes a dependency on the router:

full_scope = f"{self.resource_server_id}/{self._scope}"
if full_scope not in claims.get("scope", "").split():
 raise HTTPException(403, f"token missing required scope: {full_scope}")

app = FastAPI(dependencies=[Depends(auth)]) # authenticated everywhere
router = APIRouter(dependencies=[Depends(auth.require_scope("read"))]) # plus a scope

There is also a design decision buried here. Browser requests arriving with ALB OIDC headers pass the scope check without a scope, since the ALB already authenticated the human interactively and browser sessions have no scopes to check.

Fetching a token

The outbound half fetches tokens from Cognito and caches them until they expire.

Cache per scope, not per client. A service calling three downstream APIs holds three tokens, each carrying only the access it needs for that call.

Guard the refresh with a lock, and re-check inside it. Under concurrent load, several coroutines will notice the expired token at the same moment and all stampede Cognito’s token endpoint:

async def get_token(self, scope: str) -> str:
 cached = self._tokens.get(scope)
 if cached and time.time() < cached.expires_at:
 return cached.access_token

 if scope not in self._locks:
 self._locks[scope] = asyncio.Lock()

 async with self._locks[scope]:
 # Re-check inside the lock: another coroutine may have fetched already.
 cached = self._tokens.get(scope)
 if cached and time.time() < cached.expires_at:
 return cached.access_token
 self._tokens[scope] = await self._fetch_new_token(scope)
 return self._tokens[scope].access_token

Expiry is stored with a 60 second skew subtracted, so a token is treated as expired slightly before Cognito thinks so. Without that, a token that passes the check and then spends 400ms in flight can arrive already invalid.

Required ALB rules

If your ALB has a Cognito authentication action on its listener rule, it applies to every request on that rule. A service presenting a valid Bearer token gets a 302 redirect to the Cognito login page. It never reaches your service, so all that careful token validation never runs.

The listener needs to rules, defined in order:

Higher priority: match requests carrying an Authorization: Bearer header and forward them straight to the EC2 target group, with no authenticate action.
Lower priority: everything else gets authenticate-cognito and the normal browser login.

With respect to Gate 1, you could also consider matching on source IP instead, forwarding anything from an allowlisted CIDR. This fixes Gate 1, and Gate 2 is unchanged either way.

However, a source-IP matching rule forwards without running the authenticate action. So once you allowlist the office or VPN range, a person opening a dashboard in a browser from that network skips the login entirely and arrives with no session and no token. This is DEFINITELY NOT what you want, because we want all requests to be authenticated. This is specific to browser users, and S2S communication still works.

Also, allowlisting the VPC’s own CIDR admits in-VPC services only. A public ALB sees a laptop’s corporate or VPN egress IP. That address is nowhere near the VPC range. So a VPC-CIDR rule never admits laptops, even on VPN. If in-VPC calls work and laptops get redirected to a login page, this is why.

Gotcha! Header bypass can worsen an authentication bug

We trust browser requests based on the presence of the x-amzn-oidc-identity header, and that the browser path skips scope checks. That’s normally safe, because the ALB’s Cognito action overwrites that header on every request it processes, so a client can’t forge it.

A bypassed request doesn’t run that action, so with a header-match rule, anyone on the internet can send:

Authorization: Bearer anything
x-amzn-oidc-identity: someone@example.com

The first header triggers the bypass. The second makes your service treat the request as an authenticated browser user, which skips the scope check entirely. The Bearer value is never validated, because the ALB-header path returns before the token path is reached.

The mitigations, either of which closes it:

Strip inbound x-amzn-oidc-* headers on the bypass path, so only the ALB can ever set them.
Dont trusting identity by presence of the header alone. Verify the signed x-amzn-oidc-data JWT rather than reading x-amzn-oidc-identity as a plain string.

The second is the better fix because it removes a whole class of bugs instance of one instance of it.

Calling it from anywhere

Something that confused me initially was that curl requests from the CLI would fail. But this was because we weren’t passing in any form of authentication into the request, which made local testing or scenarios like calling a service directly from an EC2 instance, fail. I enabled this functionality with two HTTP steps that are pretty much identical in Python, Java, or a shell script:

TOKEN=$(curl -s -X POST "https://<domain>.auth.<region>.amazoncognito.com/oauth2/token" \
 -u "$CLIENT_ID:$CLIENT_SECRET" \
 -d grant_type=client_credentials \
 -d 'scope=https://api.example.com/service-a/read' | jq -r .access_token)

curl "https://api.example.com/service-a/things" -H "Authorization: Bearer $TOKEN"

A Python client is a convenience wrapper around these steps – it’s not the ONLY way to run this authorization flow. Anything that can make an HTTPS request can participate. But we might want to alleviate this by building in a seperate app client for developers and for automation purposes.

Wrapping up

This was a pretty approachable problem, especially since we already had the infrastructure in place. We were already using Cognito to authenticate browser users through the ALB, and the client-credentials flow runs against the same user pool, the same JWKS endpoint, and the same validation code. We just needed one more OAuth flow on the system already in place.