Scalability on ECS

Written by Claude Code on 2026-10-18, from the load and fault tests run on the dev environment of arglabs-workload between 16 and 18 October 2026: nine rounds, four reports, more than 24 million requests. The module is arglabs-tfmod-ecs-app; arglabs-workload is the template every ARGLabs app repository copies.

How fast an ECS Fargate service adds tasks when the load jumps, what each scaling option of the module changes, and what the app itself has to do. Every number below was measured, on the same service, with the same tests, from one client.

1. Summary

2. The test bench

The tests, each one starting from 2 tasks unless noted:

3. Why the default policies are slow

ECS publishes CPUUtilization and MemoryUtilization and the ALB publishes RequestCountPerTarget as one-minute datapoints. In these tests a minute’s datapoint reached the alarm about 3 minutes after that minute ended. Everything built on them inherits the delay:

The late data also acts after the facts. A CPU emergency alarm went back to ALARM 1 min 23 s after a load had ended, on a datapoint from 3 minutes before, and took an idle service from 2 to 4 tasks. In the first round, a memory AlarmLow still on from before a memory test let the CPU policy remove two tasks while the memory was at 76%: Application Auto Scaling scales in when every target tracking policy agrees, and one of them was reading the past.

4. Instrumenting the app

Fast scaling moves the signal into the task: the app measures itself and publishes three high-resolution CloudWatch metrics, and the module scales on 10-second alarms over them.

The contract

• Namespace and dimensions are the contract. The module injects METRICS_NAMESPACE (<scope>/<app>), METRICS_INTERVAL (10), METRICS_DIMENSIONS (ClusterName=<cluster>,ServiceName=<service>) and METRICS_BURST_THRESHOLD. Every task publishes under the same dimensions, so an alarm’s statistic over a period is a statistic over the tasks; the module’s alarms use the median (p50).
• StorageResolution 1, every 10 s, from a thread of its own.
• Publish at once on a burst. When the requests in flight reach METRICS_BURST_THRESHOLD (the module sets it from fast_scaling_inflight_threshold), the publisher does not wait for the end of its interval, at most once every 3 s: the alarm sees the spike in the period it starts in, about 10 s sooner.
• Stay quiet for the first 20 s. The sidecars and the interpreter start with a burst of CPU (81% of the task over the first interval): published, every new task would read as saturated and every deployment would scale out.
• Count the queue. A request waiting in the accept queue is load the task has not touched yet; leaving it out makes a saturated task look idle.

The module grants cloudwatch:PutMetricData to the task role with the condition cloudwatch:namespace equal to the namespace (the API takes no resource ARN). The essentials of what the example app does, as a sketch without error handling:

import json, os, threading, time, urllib.request
import boto3

NAMESPACE = os.environ.get("METRICS_NAMESPACE", "")
INTERVAL = float(os.environ.get("METRICS_INTERVAL", "10"))
BURST = int(os.environ.get("METRICS_BURST_THRESHOLD") or 0)
DIMS = [{"Name": k, "Value": v} for k, v in (p.split("=", 1)
        for p in os.environ.get("METRICS_DIMENSIONS", "").split(",") if "=" in p)]
in_flight = 0  # the request handler adds 1 on entry and subtracts 1 on exit (under a lock)


def meta(path):
    with urllib.request.urlopen(os.environ["ECS_CONTAINER_METADATA_URI_V4"] + path, timeout=2) as r:
        return json.load(r)


def publish_forever():
    cw = boto3.client("cloudwatch")
    limits = meta("/task")["Limits"]  # {"CPU": 0.25, "Memory": 512}
    started, last, prev, samples = time.monotonic(), time.monotonic(), None, []
    while True:
        time.sleep(1)
        samples.append(in_flight)  # plus the accept queue, if the server has one
        now = time.monotonic()
        burst = BURST and samples[-1] >= BURST and now - last >= 3
        if now - last < INTERVAL and not burst:
            continue
        last = now
        stats = [c for c in meta("/task/stats").values() if c]
        cpu_ns = sum(c["cpu_stats"]["cpu_usage"]["total_usage"] for c in stats)
        mem = sum(c["memory_stats"]["usage"] - c["memory_stats"]["stats"].get(
            "total_inactive_file", c["memory_stats"]["stats"].get("inactive_file", 0)) for c in stats)
        data = [{"MetricName": "RequestsInFlight", "Dimensions": DIMS, "Unit": "Count",
                 "StorageResolution": 1,
                 "StatisticValues": {"SampleCount": len(samples), "Sum": sum(samples),
                                     "Minimum": min(samples), "Maximum": max(samples)}}]
        if prev and now - started >= max(2 * INTERVAL, 20):
            cpu = 100 * (cpu_ns - prev[0]) / ((now - prev[1]) * 1e9) / limits["CPU"]
            for name, value in (("CpuUtilization", cpu),
                                ("MemoryUtilization", 100 * mem / (limits["Memory"] * 2**20))):
                data.append({"MetricName": name, "Dimensions": DIMS, "Unit": "Percent",
                             "StorageResolution": 1, "Value": round(min(value, 100), 2)})
        prev, samples = (cpu_ns, now), []
        cw.put_metric_data(Namespace=NAMESPACE, MetricData=data)


if NAMESPACE:
    threading.Thread(target=publish_forever, daemon=True).start()

Any language works: the contract is the three names, their units and the dimensions. An app that publishes nothing leaves the alarms without data (notBreaching), so nothing happens and nothing breaks; for that app, keep fast scaling off and the CPU emergency step on.

What it costs, at us-east-1 list prices: one PutMetricData call per task every 10 s is about 259,000 calls a month, USD 2.60 per task-month at USD 0.01 per 1,000; plus three custom metrics and three high-resolution alarms per service and region, USD 1.80 a month. That is about what a 0.25 vCPU Spot task costs; a 30 s interval divides the calls by three, and the alarms then use 30 s periods.

5. The options, one by one

Module inputs of arglabs-tfmod-ecs-app, the default, the value in arglabs-workload (dev), and what each one did in the tests.

Fast scaling

Policies on AWS metrics

Around the scaling: load balancer and deployments

6. Fast scaling on and off, measured

The same three tests, run with enable_fast_scaling on and off in dev, on the same app image and the same configuration: only the option changed between the two rounds.

Nothing failed with fast scaling off: this app queues instead of refusing, so a shortage of tasks shows as latency and throughput, not as errors, as long as it stays under the client’s timeout. An app with tighter timeouts, or one that refuses connections, would turn those minutes into errors. The price of fast scaling is capacity used sooner: the same 10 tasks, minutes earlier. The same option measured twice, on two app versions, moved by up to 11 s for the first scale-out and 33 s for 10 healthy tasks: noise well under the difference above.

7. Inside the task: what the app had to change

The spike of 600 against 2 tasks, the same test after each change. The first two rows are the first rounds, without fast scaling; from T1 on, fast scaling is on.

What each step taught, for any app behind an ALB:

• Check the listen backlog and keep-alive before a load test. Python’s HTTPServer listens with a backlog of 5 and answers HTTP/1.0, so the ALB opens a connection per request; at about 470 new connections a second per task the queue overflowed, the kernel dropped SYNs, and the ALB answered 504 after its 10 s connect timeout. No target ever went unhealthy.
• Unlimited concurrency is worse than a queue. With a backlog of 1024, the ALB put all 600 requests inside the 2 tasks: about 300 Python threads per task on 0.25 vCPU fought over the interpreter, each request cost twice the CPU, and the 2 s health check timed out mid-spike. The old backlog of 5 had been admission control by accident.
• Keep the accepting thread fed. With 8 worker threads, the one accepting connections waited behind them and new connections sat up to 10 s unaccepted. One worker thread, with requests that sleep handed to threads of their own, removed it.
• Turn off Nagle with keep-alive. Headers and body in two writes waited for the ALB’s delayed ACK: 40 ms on every answer. TCP_NODELAY, then one write per answer.
• Drain on SIGTERM, and let the ALB drain. Deploys, scale-in and Spot interruptions stop tasks with SIGTERM; the app keeps accepting until the kernel’s queue is empty, and deregistration_delay gives the requests in flight 5 s.
• Measure CPU per request where it runs. A local profile on Python 3.9 blamed indented JSON for 30% of a request; on Python 3.14 and later, the image’s, indent= uses the C encoder too and costs about 0.5 µs. An unpinned base image changes the CPU per request with no code change.
• Watch the file descriptor limits of old APIs. select.select() fails above 1,024 descriptors: 3000 slow requests made every request fail, health checks included. poll() has no such limit.

Capacity with the service fixed at 10 tasks, req/s at 300, 400 and 600 in flight: 2,390, 3,083 and 3,537 in the first round; 2,404, 3,205 and 4,679 with the final code. Below 300 in flight the throughput is the concurrency divided by the network latency, the same in every version.

8. Traps

• A step alarm still in ALARM fires again. Setting the desired count by hand after a test was undone by an alarm that had not gone back to OK. Wait for the alarms first.
• Late datapoints scale the next test. With fast scaling off, wait about 4 minutes after a load before measuring anything else, or the previous load scales it.
• Deployment alarms see the past too. A deploy started right after a flood of 40 s requests was rolled back: the response time alarm caught the flood two minutes late, inside the bake time. Deploy once the alarms are OK.
• HealthyHostCount misses short outages. The ALB publishes it with 2 samples a minute: 40 s without a healthy target never showed in the metric, a full minute did.
• Logs lag when the CPU is saturated. Fluent Bit shares the task’s 0.25 vCPU: only 13% of the lines reached CloudWatch in their own minute, the rest up to 5 minutes later. None were lost; filter by the @timestamp inside the JSON, not by CloudWatch’s.
• A new task needs about a minute. About 19 s from creation to the image pull, 30 to 60 s of pulls (the Fluent Bit image is 113 MB compressed), then two health checks: smaller images are the next gain.
• Spot interruptions are cheap once the task drains. Five real ones during the tests. The one in the middle of a spike, before the 5 s drain, cut 21 requests in flight; the other four cost none: the replacement started a second after the notice and was healthy in 40 to 50 s.

9. The configuration that came out of it

The module inputs as arglabs-workload sets them in dev, through its locals:

module "ecs_app" {
  source = "git::https://bitbucket.org/arglabs/arglabs-tfmod-ecs-app.git//terraform?ref=main"
  # ...
  min_capacity_value         = 2
  max_capacity_value         = 10
  cpu_target_value           = 70
  mem_target_value           = 70
  request_count_target_value = 1000 # requests a minute per task: size it from a measured task

  # Fast scaling: the app publishes CpuUtilization, MemoryUtilization and RequestsInFlight
  fast_scaling_metrics_namespace      = "aio/workload"
  fast_scaling_metrics_interval       = 10
  fast_scaling_evaluation_periods     = 1
  fast_scaling_datapoints_to_alarm    = 1
  fast_scaling_statistic              = "p50"
  fast_scaling_inflight_threshold     = 8
  fast_scaling_step_percent           = 200
  fast_scaling_saturated_step_percent = 400
  fast_scaling_cooldown               = 60
  fast_scaling_target_tracking        = true

  # The steps on late metrics only where fast scaling is off
  enable_emergency_step_scaling     = false
  emergency_request_count_threshold = 0

  # Draining, and deployments that roll back on errors
  deregistration_delay            = 5
  deployment_bake_time_in_minutes = 5
  deployment_module_alarms        = ["http_5xx", "target_response_time"]
  http_5xx_threshold              = 10
  target_response_time_threshold  = 1
}

For an app that cannot publish the metrics: leave fast_scaling_metrics_namespace empty, keep enable_emergency_step_scaling on, and set request_count_target_value from a measured task; expect 4 to 6 minutes to the first scale-out.

10. Method and limits

• One client, in Brazil, about 120 ms from the service: client-side throughput includes that latency; the ALB’s TargetResponseTime gave the server’s side.
• Fargate Spot on Graviton2, different days and hours: the first round’s ramp numbers came partly from that day’s hardware. Its image, run again on the later tasks, did 2,308, 2,540 and 2,584 req/s at 300, 400 and 600 in flight, against 2,390, 3,083 and 3,537 that day.
• One run per configuration in most tests; the variation between runs of the same code is in the text where it was measured.
• Scale-in was not measured after the first round: between tests the desired count went back to 2 by hand, once the scaling alarms were OK.
• Dev only: stg and prd run the same code with other values (prd keeps 2 on-demand tasks before Spot).