Service-to-Service Auth with Cognito Scopes

I’m building tools that help scientists run their day-to-day work. I work in a heavily regulated environment, subject to both U.S. and European digital policy requirements. The scenario we face regularly is: two services talk to each other. Then three. A dashboard calls a backend API, that API calls another one, a nightly job calls all of them. Somewhow, we have to answer how one service proves to another that it is allowed to make the call.

The first way I addressed this was to use Internal Bearer Tokens. Basically, you generate a long random string, store it in the AWS Secrets Manager, inject it into every service as an environment variable, have each service check inbound requests for it.

I replaced that solution with Cognito’s OAuth2 client-credentials flow. Here each service carries its own identity and each endpoint declares the scope it requires. We didn’t need to define any new infrastructure for this, since Cognito was already authenticating browser users through the ALB, so the service-to-service (S2S) path ended up reusing the same user pool, the same JWKS endpoint, and the same validation code that was already there.

Why a shared bearer token was not viable

The bearer token approach was really easy, but it wasn’t a viable approach in the long run, for a few reasons:

No expiry. A static token has no TTL. If it leaks through a log line, a memory dump, or a compromised container, it stays valid until you manually rotate it and run a redeploy of every service holding it. There is no safe window during rotation where both the old and new values work.

No identity. When service B receives the token, it can verify the value matches what it was given, but it can’t tell who sent it. Every caller is indistinguishable, so you can only observe that the token was used and not who made the call.

No scope. The receiving service has no way to enforce what the caller may do. Either it accepts the token and grants full access, or it rejects it. There’s no way to express “service A may read but not write”.

Shared blast radius. The same secret works everywhere. One compromised service exposes the key to every other service. This is a huge vulnerability.

Comparing these two solutions against each other, we have:

Being able to attribute traffic to a caller enhances the utility of logs, which is most of the subject of my next post.

Two gates

There are two authentication and authorization gates in this architecture:

Gate 1 is the ALB. Everything in our system is either accesible from outside our VPC via a VPN connection through the ALB, or has ingress/egress restricted to within the VPC. The ALB decides only how a request gets in: either this is an authenticated browser user who must complete an interactive Cognito login, or it’s a caller to hand straight to the service. The ALB knows nothing about your application’s permissions.

Gate 2 is the service. It validates identity and enforces scope. This is where the real authorization check happens for service-to-service communication.

An incoming request has to clear both the authentication and the authorization. A browser user who hasn’t been authenticated gets a 302 status to the Cognito login page via Gate 1 and never reaches the dashboard or API, whereas a service with a valid token but the wrong scope passes Gate 1 but receives a 403 at Gate 2.

caller ──▶ [ Gate 1: ALB ] ──▶ [ Gate 2: service ] ──▶ your handler
            browser or machine?   who are you, and what may you do?
              │ browser,            │ no or invalid credential
              │ not logged in       │ valid token, wrong scope
              ▼                     ▼
         302 → Cognito login     401 / 403

What services can call what

Below is an example diagram of allowed communication. Solid arrows are requests, and each one is labeled with the scope it requires.

graph LR
  Human(Browser user)
  Robot(Laptop / CI job)
  ALB{ALB}
  Dash[Dashboard]
  SvcA[Service A]
  SvcB[Service B]
  SvcC[Service C]
  Cognito(Cognito)

  Human -->|"session cookie, no scope"| ALB
  Robot -->|"Authorization: Bearer"| ALB

  ALB -->|"x-amzn-oidc-identity"| Dash
  ALB -->|"Bearer passthrough"| SvcA

  Dash -->|"service-a/read"| SvcA
  Dash -->|"service-b/read"| SvcB
  SvcA -->|"service-b/read"| SvcB
  SvcA -->|"service-c/write"| SvcC

  Cognito -.->|"token: service-a/read, service-b/read"| Dash
  Cognito -.->|"token: service-b/read, service-c/write"| SvcA

  classDef human fill:#e8f0fe,stroke:#4285f4
  classDef machine fill:#e6f4ea,stroke:#34a853
  class Human human
  class Robot machine

A browser user reaches the dashboard with no scope at all, because the ALB already authenticated them interactively and the dashboard is the thing they’re allowed to look at. The dashboard’s own token is narrower than service A’s: it can read service A and service B, but nothing grants it write access to service C. If the dashboard is compromised, the write path isn’t reachable from there.

The permissions are a table, which makes adding a consumer easy. You don’t have to change any of the actual code:

Cognito resource servers, custom scopes, and app clients

The resource server defines capabilities

A resource server represents a service that exposes capabilities. It has two name-like fields that do very different jobs:

# terraform code

resource "aws_cognito_resource_server" "service_b" {
  user_pool_id = var.cognito_user_pool_id

  # The scope prefix. Opaque string, conventionally a URL. Immutable in practice.
  identifier = "https://api.example.com/service-b"

  # Console display name only. Nothing references it.
  name = "service-b"

  scope {
    scope_name        = "read"
    scope_description = "Read access to service-b"
  }

  scope {
    scope_name        = "write"
    scope_description = "Write access to service-b"
  }

  scope {
    scope_name        = "admin"
    scope_description = "Administrative access to service-b"
  }
}

The identifier field becomes the prefix of every scope the resource server defines, so the three scopes above are represented as

https://api.example.com/service-b/read
https://api.example.com/service-b/write
https://api.example.com/service-b/admin

That full string is what a client requests, what appears in the JWT’s scope claim, and what a service compares against, whereas name is just a display label in the console.

Only custom scopes work for machine-to-machine

The client-credentials flow works only with custom scopes from a resource server. I defined these scope manually in my FastAPI applications, and decorated the relevant endpoints with their respective scopes. The built-in OpenID scopes (openid, email, profile, aws.cognito.signin.user.admin) are for user-facing flows (e.g. logins). Every machine caller needs at least one resource server to exist, even if the service it calls has exactly one capability.

The app client selects a subset

An app client represents a caller of a service. One per calling service, with allowed_oauth_scopes listing every downstream capability it may request:

resource "aws_cognito_user_pool_client" "service_a_m2m" {
  name         = "service-a-m2m"
  user_pool_id = var.cognito_user_pool_id

  generate_secret                      = true
  allowed_oauth_flows_user_pool_client = true
  allowed_oauth_flows                  = ["client_credentials"]

  # Full scope strings, exactly as composed above.
  allowed_oauth_scopes = [
    "https://api.example.com/service-b/read",
    "https://api.example.com/service-c/write",
  ]

  # Cognito validates that each scope exists, so the resource servers
  # must be created first. Terraform will not infer this ordering.
  depends_on = [
    aws_cognito_resource_server.service_b,
    aws_cognito_resource_server.service_c,
  ]
}

Service A is granted read on service B, and write on service C. Service A can’t write to service B even though that scope exists, because its app client never lists it as a viable permissions. Scopes model capabilities, not consumers. Service B declares that reading, writing, and administering are things that can be done to it (this is defined both in the FastAPI code via decorated endpoints, and in the Cognito resource server). But Service B doesn’t say anything about who can take those actions. Consumers get assigned action permissions through their app client’s allowed-scopes list.

This separation lets you onboard a caller without disrupting the callee. A new consumer is one app client and one secret. Service B does not redeploy, does not learn the new caller’s name, and does not grow a config entry. Its code already says require_scope("read").

Two headaches I ran

An omitted scope parameter is not the same as no scopes. If a token request leaves scope off, Cognito issues a token carrying every scope the app client is allowed. This means that every downstream call then presents maximum privilege. Basically, everything is open, unless specified otherwise.

Removing a scope is harder than adding one. Cognito won’t let you delete a scope from a resource server while an app client still lists it in allowed_oauth_scopes. The order is: remove it from every app client, apply, then remove it from the resource server. This was actually quite annoying, and I couldn’t figure out initially why changing and removing scopes was such a headache.

Validating a token

The receiver has to check four things:

1. The signature, against Cognito’s public JWKS for the pool.
2. The issuer, so a token from some other pool isn’t accepted.
3. token_use == "access", so an ID token can’t be substituted for an access token.
4. The scope claim contains the specific scope the endpoint requires.

def _decode_token(self, token: str) -> dict:
    signing_key = self._jwk_client.get_signing_key_from_jwt(token)

    claims = jwt.decode(
        token,
        signing_key.key,
        algorithms=["RS256"],
        issuer=self.issuer,
        # Client-credentials tokens carry `client_id` instead of `aud`,
        # so skip audience verification and check token_use instead.
        options={"verify_aud": False},
    )

    if claims.get("token_use") != "access":
        raise HTTPException(401, "wrong token type")

    return claims

Client-credentials tokens have no aud claim – they carry client_id , so leaving audience verification on rejects every valid S2S token with a confusing error. Turning it off here is valid, but only because token_use and the issuer are being checked instead.

Scope enforcement then becomes a dependency on the router:

full_scope = f"{self.resource_server_id}/{self._scope}"
if full_scope not in claims.get("scope", "").split():
    raise HTTPException(403, f"token missing required scope: {full_scope}")

app    = FastAPI(dependencies=[Depends(auth)])                       # authenticated everywhere
router = APIRouter(dependencies=[Depends(auth.require_scope("read"))])  # plus a scope

Fetching a token

The outbound half fetches tokens from Cognito and caches them until they expire.

Cache per scope, not per client. A service calling three downstream APIs holds three tokens, each carrying only the access it needs for that call.

Guard the refresh with a lock, and re-check inside it. Under concurrent load, several coroutines will notice the expired token at the same moment and all stampede Cognito’s token endpoint:

async def get_token(self, scope: str) -> str:
    cached = self._tokens.get(scope)
    if cached and time.time() < cached.expires_at:
        return cached.access_token

    if scope not in self._locks:
        self._locks[scope] = asyncio.Lock()

    async with self._locks[scope]:
        # Re-check inside the lock: another coroutine may have fetched already.
        cached = self._tokens.get(scope)
        if cached and time.time() < cached.expires_at:
            return cached.access_token
        self._tokens[scope] = await self._fetch_new_token(scope)
        return self._tokens[scope].access_token

Required ALB rules

If your ALB has a Cognito authentication action on its listener rule, it applies to every request on that rule. A service presenting a valid Bearer token gets a 302 redirect to the Cognito login page. It never reaches your service, so all that careful token validation never runs.

The listener needs to rules, defined in order:

1. Higher priority: match requests carrying an Authorization: Bearer header and forward them straight to the EC2 target group, with no authenticate action.
2. Lower priority: everything else gets authenticate-cognito and the normal browser login.

Gotcha! Source-IP bypass breaks browser login

With respect to Gate 1, you could also consider matching on source IP instead, forwarding anything from an allowlisted CIDR. This fixes Gate 1, and Gate 2 is unchanged either way.

However, a source-IP matching rule forwards without running the authenticate action. So once you allowlist the office or VPN range, a person opening a dashboard in a browser from that network skips the login entirely and arrives with no session and no token. This is DEFINITELY NOT what you want, because we want all requests to be authenticated. This is specific to browser users, and S2S communication still works.

Also, allowlisting the VPC’s own CIDR admits in-VPC services only. A public ALB sees a laptop’s corporate or VPN egress IP. That address is nowhere near the VPC range. So a VPC-CIDR rule never admits laptops, even on VPN. If in-VPC calls work and laptops get redirected to a login page, this is why.

Gotcha! Header bypass can worsen an authentication bug

We trust browser requests based on the presence of the x-amzn-oidc-identity header, and that the browser path skips scope checks. That’s normally safe, because the ALB’s Cognito action overwrites that header on every request it processes, so a client can’t forge it.

A bypassed request doesn’t run that action, so with a header-match rule, anyone on the internet can send:

Authorization: Bearer anything
x-amzn-oidc-identity: someone@example.com

The first header triggers the bypass. The second makes your service treat the request as an authenticated browser user, which skips the scope check entirely. The Bearer value is never validated, because the ALB-header path returns before the token path is reached.

The mitigations, either of which closes it:

• Strip inbound x-amzn-oidc-* headers on the bypass path, so only the ALB can ever set them.
• Dont trusting identity by presence of the header alone. Verify the signed x-amzn-oidc-data JWT rather than reading x-amzn-oidc-identity as a plain string.

The second is the better fix because it removes a whole class of bugs instance of one instance of it.

Calling it from anywhere

Something that confused me initially was that curl requests from the CLI would fail. But this was because we weren’t passing in any form of authentication into the request, which made local testing or scenarios like calling a service directly from an EC2 instance, fail. I enabled this functionality with two HTTP steps that are pretty much identical in Python, Java, or a shell script:

TOKEN=$(curl -s -X POST "https://<domain>.auth.<region>.amazoncognito.com/oauth2/token" \
  -u "$CLIENT_ID:$CLIENT_SECRET" \
  -d grant_type=client_credentials \
  -d 'scope=https://api.example.com/service-a/read' | jq -r .access_token)

curl "https://api.example.com/service-a/things" -H "Authorization: Bearer $TOKEN"

A Python client is a convenience wrapper around these steps – it’s not the ONLY way to run this authorization flow. Anything that can make an HTTPS request can participate. But we might want to alleviate this by building in a seperate app client for developers and for automation purposes.

Wrapping up

This was a pretty approachable problem, especially since we already had the infrastructure in place. We were already using Cognito to authenticate browser users through the ALB, and the client-credentials flow runs against the same user pool, the same JWKS endpoint, and the same validation code. We just needed one more OAuth flow on the system already in place.