I thought it’d be super useful if we could do the same thing with cURL. That is, authenticate cURL requests directly from the browser’s cookie store. So I’ve just published the first release of crul (pronounced “cruel”). Let me show you how it works:

Say you’re logged into a website in your web browser, and you want to access that site through cURL. crul makes that super easy:

npx @kieranhunt/crul --browsers firefox --url "https://news.ycombinator.com"
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by crul.
# Edit at your own risk.


#HttpOnly_news.ycombinator.com	FALSE	/	TRUE	0	user	kieranhunt&SECRET

That spits out a Netscape HTTP Cookie File. Perfectly compatible with cURL. Avoid the intermediate step of saving the file to disk by piping the output directly into cURL:

curl \
  -b <(npx @kieranhunt/crul --browsers firefox --url "https://news.ycombinator.com") \
  --silent \
  "https://news.ycombinator.com/upvoted?id=kieranhunt" \
  | htmlq '.titleline > a' --text
  
# Show HN: C++ AWS MSK IAM Auth Implementation – Goodbye Kafka Passwords
# Two Starkly Similar Novels and the Puzzle of Plagiarism
# Browning Fever: A story of fandom, literary societies, and impenetrable verse
# Alphabet’s Sidewalk Labs seeks share of property taxes for Toronto smart city
# Cipherli.st – Strong Ciphers for Apache, Nginx and Lighttpd

crul’s CLI parameters map one-to-one to sweet-cookie’s GetCookiesOptions type. I’d like to make it so that anything you can do with sweet-cookie you can also do with crul.

Find it @ KieranHunt\/crul on GitHub or @kieranhunt/crul on npm.

1. First, remember the name of the service that you’re looking for. ⌘ + f for it on the docs page. I usually remember to remove spaces and that its all lowercase. Then find it in the sidebar.
2. After that I expand that section of the sidebar (with my mouse 😱) and then ⌘ + f again for the Construct I’m looking for. Then I click on the link to go to the Construct’s documentation.

To speed up this process, I’ve created CDK Search:

It’s a continously updated search index of all L1 and L2 CDK constructs found in aws-cdk-lib (archive). It’s super fast, offline friendly* and supports keyboard navigation.

My workflow is now:

1. Go to CDK Search, type some portion of the construct’s name. Press ⏎ to open the docs.

That’s it. It’s a super focused experience. I’d love to hear if you find it useful. Or even better, what you’d like to see added next.

*⌘ + s the page as HTML and you can view a (stale) offline copy any time!

It uses the same technique I described in Every AWS SDK Waiter so do give that a read if you’re interested. For more on paginators, read Stop writing AWS SDK pagination loops that break.

I’ve used Jest’s snapshot tests for years. And now there’s a way to run snapshot tests on the JVM 🙌. All thanks to selfie.

🗒️ Tests performed using JDK21 (Corretto), Kotlin v2.2.20, and selfie-runner-junit5 v2.5.0.

Inline snapshots

My favourite kind of snapshot tests are inline ones. I love grouping the snapshots with the code making the assertion. Selfie supports these just fine.

@Test
fun `test inline snapshot`() {
    Selfie.expectSelfie(listOf(1, 2, 3).toString()).toBe_TODO()
}

@Test
fun `test inline snapshot`() {
    Selfie.expectSelfie(listOf(1, 2, 3).toString()).toBe("[1, 2, 3]")
}

Disk snapshots

Disk snapshots work exactly as you’d expect.

@Test
fun `test disk snapshot`() {
    Selfie.expectSelfie(listOf("a", "b", "c").toString()).toMatchDisk_TODO()
}

@Test
fun `test disk snapshot`() {
    Selfie.expectSelfie(listOf("a", "b", "c").toString()).toMatchDisk()
}

╔═ testDiskSnapshot ═╗
[a, b, c]
╔═ [end of file] ═╗

Updating snapshots

As you iterate on your code, you’ll need to update your existing snapshots to match new expectations. Selfie provides a bunch of different ways to do this.

Delete the snapshot and add _TODO back

The is the most basic way of updating snapshots. You just delete the existing snapshot and add _TODO back to the tests. But this will quickly become laborious and time consuming.

So you turn this:

@Test
fun `test inline snapshot`() {
    Selfie.expectSelfie(listOf(1, 2, 3).toString()).toBe("[1, 2, 3]")
}

back in to this:

@Test
fun `test inline snapshot`() {
    Selfie.expectSelfie(listOf(1, 2, 3).toString()).toBe_TODO()
}

And then re-run the tests.

Via the selfie/SELFIE environment variable / system propety.

Probably my favourite method is to just re-run the test suite and tell selfie to update all snapshots.

./gradlew test -Pselfie=overwrite
# or
SELFIE=overwrite ./gradlew test

I like this method because it matches how I update Jest snapshots (npm run test -- --updateSnapshot). You can use your version control to see which snapshots have changed and revert the snapshots back if they don’t look right.

With the //selfieonce magic comment

Adding this comment into the test file will cause selfie to overwrite the snapshots in the file and then remove the comment.

With the //SELFIEWRITE magic comment

The same as above but the comment will stay there. Subsequent test runs will overwrite the snapshots.

Use SelfieSettingsAPI to write custom formatters

In much the same way that AssertJ lets your write custom assertions, selfie lets you write custom snapshot serializers. In the following example, I’ve written a serializer that makes string lists look like markdown.

object CustomSelfie : SelfieSettingsAPI() {
    fun expectSelfie(response: List<String>): StringSelfie {
        return expectSelfie(response) {
            Snapshot.of(it.joinToString(
                prefix = "\n", 
                separator = "\n", 
                postfix = "\n"
            ) { item -> "- $item" })
        }
    }
}

@Test
fun `test custom serializer`() {
    CustomSelfie.expectSelfie(listOf("a", "b", "c")).toBe_TODO()
}

@Test
fun `test custom serializer`() {
    CustomSelfie.expectSelfie(listOf("a", "b", "c")).toBe("""
- a
- b
- c
""")
}

To build that table, I’m relying on a technique I learned from Simon Willison. It works as follows:

1. Using GitHub actions, run a workflow daily.
2. In the workflow, git clone the AWS SDK for Java repository.
3. Use a script to find all waiter files and compile a markdown table with their configuration. Make the script update the repo’s README.
4. Back in the GitHub Action’s worklow, commit the changes (if there are any) before pushing.

Read the entire workflow in aws-sdk-waiters/.github/workflows/daily-aws-sdk-check.yml at main · KieranHunt/aws-sdk-waiters on GitHub.

1. Use AWS SDK Waiters
2. Write custom waiters
3. Typescript waiters are a bit weird

💃 This post is dynamic 💃
Each configuration value uses a simulator and timeline for visualizations. Many of the retry behaviours depend on random values for things like jitter. Reload this page to see how randomisation affects them.

🎮 This post includes a simulator 🎮
Scroll to the bottom of this page to use the complete simulator. It lets you configure every parameter in WaiterOverrideConfiguration and see how it affects the retry behaviour.

Introduction

WaiterOverrideConfiguration (docs, archive) is how you configure the timeout and retry behaviour for AWS SDK Waiters on the JVM.

Each AWS service defines their own WaiterOverrideConfiguration. These are configured per-waiter method. That is, the waiter for a CloudFront’s DistributionDeployed is going to look quite different to the waiter for ECS’s TasksRunning. For example, I dug into the EC2 waitUntilInstanceRunning method and pulled out the following WaiterOverrideConfiguration:

WaiterOverrideConfiguration.builder()
  .maxAttempts(40)
  .waitTimeout(null)
  .backoffStrategyV2(
    BackoffStrategy.fixedDelayWithoutJitter(
      /* delay */ 15.seconds.toJavaDuration()
    )
  )
  .build()

In the rest of this post, I dig in to what each of those configuration values means and how best to pick between them.

Just before I start, I’m going to show the behaviour of waiters using different configuration values. It helps to have a certain resource in mind while going through these examples. I won’t use any resource in particular, but maybe you could think of:

• Waiting for an SSM Command Invocation to complete
• Waiting for a CloudWatch Insights Query to complete, or
• Waiting for a Kinesis Stream to be created.

In the following examples the resource is, by default, set to reach its desired state after 10 seconds. If the waiter hasn’t timed out or reached its maximum attempt threshold, it will transition to success

maxAttempts

The maxAttempts value sets an upper bound on the number of times the waiter will check the state of the resource. In the below example, I’ve set maxAttempts to low enough that the waiter will never finish.

After two checks of the resource, the waiter gives up.

With the rest of the configuration parameters available for a waiter, I don’t actually think its necessary to constrain an upper bound on attempts. Doing so means you have to do some complicated maths (or use my simulator below) to figure out the right upper value to pick. Instead, trusting waitTimeout and picking a sensible delay between API calls should be enough.

waitTimeout

The waitTimeout value configures how long you’re willing to wait for the resource to transition into the desired state. This is probably the most important value to configure in your waiter.

Some resources, like the SSM Commands Invocations in my previous post, should have a timeout value that depends on what the command is doing. If your command includes a 5-minute sleep then your timeout should probably be configured for something a little bit more than 5 minutes. SSM has some delay in delivering the command to the target instances, so you’ll need to factor that in too.

Other resources, like an EC2 instance going into running, mostly depend on the AWS service and so can be generically defined. It’s probably fine to leave it up to whatever default AWS picked.

backoffStrategyV2

retryImmediately

The simplest backoff strategy is retryImmediately. It’s the for loop of backoff strategies.

The waiter iterates, checking the state of the resource, over and over again, until, at some point, the resource reaches the desired state. The next check completes the waiter.

In the following example, I’ve shrunk the resource state change delay to 1 second. This keeps it within bounds for the simulator.

fixedDelayWithoutJitter

The first real “strategy” is fixedDelayWithoutJitter. This does exactly what you’d think. It waits for the specified delay between checks of the resource.

This seems to be the default backoff strategy in the SDKs. At least, I’ve yet to see one using something else.

fixedDelay

Fixed delay has a little secret. It includes jitter!

With jitter, fixed delay waits between 0 seconds and the delay amount before re-checking the resource. This means that you always get at least the same number of checks of the resource as you would with fixedDelayWithoutJitter. Often you get more.

exponentialDelayWithoutJitter

Exponential delay, unlike fixed delay, introduces more and more wait time between checks of the resource. The algorithm looks like this:

delay = min(baseDelay × 2 ^ (attempt - 1), maxDelay)

With a large enough maxDelay, and a resource that transitions state towards the second half of the waitTimeout, you can expect substantial periods of waiting. In the following example, the last wait took 8 seconds which is more than half of the entire waiter period.

Since the algorithm picks the minimum between the baseDelay and maxDelay, you can actually reproduce the fixedDelay output by matching the two delays:

exponentialDelay

Full jitter combines the jitter implementation as seen in fixedDelay with an exponential delay.

exponentialDelayHalfJitter

And finally, half jitter gives you jitter somewhere between half the full value and the full value. This gives you something closer to the behaviour of not having jitter while still keeping the jitter safety.

Simulator

Play around with the simulator by tweaking any of its values below.

I’ll start here with an example using SSM and waiting on a Command Invocation. That’s the resource you get back when you call SendCommand. It’s just like the examples in the Use AWS SDK Waiters post which focuses on the JVM.

A successful waiter

import {
  SendCommandCommand,
  SSMClient,
  waitUntilCommandExecuted,
} from "@aws-sdk/client-ssm";

const ssmClient = new SSMClient();

const sendCommandOutput = await ssmClient.send(new SendCommandCommand({
  InstanceIds: [instanceId],
  DocumentName: "AWS-RunShellScript",
  Parameters: {
    "commands": [
      /* https://kieran.casa/starting-bash-scripts/ */
      "#!/usr/bin/env bash",
      "set -o xtrace",
      "set -o errexit",
      "set -o nounset",
      "set -o pipefail",
      "echo 'Hello, World!'",
      "exit 0"
    ]
  }
}));

const waiterResult = await waitUntilCommandExecuted({ client: ssmClient, maxWaitTime: 30 }, {
  CommandId: sendCommandOutput.Command?.CommandId,
  InstanceId: instanceId
});

const getCommandInvocationCommandOutput = waiterResult.reason as GetCommandInvocationCommandOutput;
  
assert.equal(getCommandInvocationCommandOutput.Status, "Success");

Here’s where the javascript waiters start to drift from the Java ones. First, you must import the wait function directly. It’s not a function on the ssmClient.

You pass in the client and maxWaitTime as fields in a object for the first parameter. maxWaitTime is how long you’re willing to wait (in seconds) for the operation to complete.

Just to linger on that maxWaitTime for a little bit. The Java-based waiters always have a default wait time. AWS decides for them how long waiters should wait. But there are loads of occasions where it makes sense for you to override that value. Like for resources where you control how long that take to run—and so you know best. Or is situations where you have a finite time budget. SSM Command Invocations is a great example of when you know better than AWS how long you expect them to take.

minDelay and maxDelay are also accepted along side maxWaitTime but are optional. The waiter doesn’t let you configure a retry strategy. Instead it always uses exponential backoff. If you want to get back to a fixed wait time back-off, just set minDelay and maxDelay to the same values.

You get back a WaiterResult and in the reason property is the result of the last polling operation. In our case its a GetCommandInvocationCommandOutput. But it’s not strongly typed… Instead its any. So you need to cast it into GetCommandInvocationCommandOutput if you want to reach in to any of its fields.

A failed waiter

Now here’s what a waiter looks like when it fails. That is, when the resource enters a state where it could never possibly reach the desired state.

You’ll see that I’ve used Node’s assertion library in the rest of this post. Waiters are particularly useful during integration-style tests and so you’ll often see assertions paired with waiters this way.

const sendCommandOutput = await ssmClient.send(new SendCommandCommand({
  InstanceIds: [instanceId],
  DocumentName: "AWS-RunShellScript",
  Parameters: {
    "commands": [
      "#!/usr/bin/env bash",
      "exit 1" /* ← bound to fail */
    ]
  }
}));

await assert.rejects(
  waitUntilCommandExecuted({ client: ssmClient, maxWaitTime: 30 }, {
    CommandId: sendCommandOutput.Command?.CommandId,
    InstanceId: instanceId
  }),
  (error: Error) => {
    assert.ok(error.name === "Error");

    const parsedMessage = JSON.parse(error.message);

    assert.equal(parsedMessage.state, "FAILURE");
    assert.ok("200: OK" in parsedMessage.observedResponses);

    assert.partialDeepStrictEqual(parsedMessage.reason as GetCommandInvocationCommandOutput, {
      Status: "Failed",
      StandardErrorContent: "failed to run commands: exit status 1",
      StatusDetails: 'Failed',
      // And all of the rest of the output fields
    });

    return true;
  }
);

So here’s the next bit of weirdness. The AWS SDK hides a bunch of information in a JSON string inside the error’s message property. Hmm…

Inside a failed waiter message we find:

• A state field with the string FAILURE.
• An observedResponses object where the keys are the response HTTP status codes and the values are the count.
• A reason object which is the response from the last call. Again you’ll want to cast into the actual output type.

When using a waiter like this, that reason information will probably be very helpful when debugging.

A waiter that times out

To make a waiter time out, we can just make the SSM Command sleep for longer than the waiter is willing to wait.

const sendCommandOutput = await ssmClient.send(
  new SendCommandCommand({
    InstanceIds: [instanceId],
    DocumentName: "AWS-RunShellScript",
    Parameters: {
      commands: [
        "#!/usr/bin/env bash",
        "sleep 60" /* ← that'll sleep for 1 minute */,
        "exit 0",
      ],
    },
  }),
);

await assert.rejects(
  waitUntilCommandExecuted(
    {
      client: ssmClient,
      /**
       * `maxWaitTime` is set to something well below the 60 second sleep.
       * This is guaranteed to fail.
       */
      maxWaitTime: 5,
    },
    {
      CommandId: sendCommandOutput.Command?.CommandId,
      InstanceId: instanceId,
    },
  ),
  (error: Error) => {
    assert.ok(error.name === "TimeoutError");

    const parsedMessage = JSON.parse(error.message);

    assert.equal(parsedMessage.state, "TIMEOUT");
    assert.equal(parsedMessage.reason, "Waiter has timed out");
    assert.ok("200: OK" in parsedMessage.observedResponses);

    return true;
  },
);

Again you’ll see that we need to parse information out of JSON in the error message.

Unfortunately that’s all the information that you’re given when a waiter times out. It would’ve been great if it returned the value of the last response it got before timing out.

Racing waiters

Finally, waiters can be aborted if you decide that you no longer want to wait for them.

Waiters accepts an (optional) abort signal parameter. You can create the signal by first creating an AbortController and then passing along the signal. An abort controller allows you to send a cancellation message to the waiter thread.

In this post I use Promise.race. It accepts an array of promises and will wait for the first promise to resolve (or reject). My waiters are configured with the same timeout (PT30S) but I expected Command #2 to always finish first.

const sendCommandOutputOne = await ssmClient.send(
  new SendCommandCommand({
    InstanceIds: [instanceId],
    DocumentName: "AWS-RunShellScript",
    Parameters: {
      commands: [
        "#!/usr/bin/env bash",
        "sleep 60" /* ← 🥈 always going to lose */,
        "exit 0",
      ],
    },
    Comment: "Command #1",
  }),
);

const sendCommandOutputTwo = await ssmClient.send(
  new SendCommandCommand({
    InstanceIds: [instanceId],
    DocumentName: "AWS-RunShellScript",
    Parameters: {
      commands: [
        "#!/usr/bin/env bash",
        "sleep 5" /* ← 🥇 always going to win */,
        "exit 0",
      ],
    },
    Comment: "Command #2",
  }),
);

const commandOneWaiterController = new AbortController();
// Note that we don't _await_ the waiter. We just want the Promise.
const commandOneWaiterPromise = waitUntilCommandExecuted(
  {
    client: ssmClient,
    maxWaitTime: 30,
    abortSignal: commandOneWaiterController.signal,
  },
  {
    CommandId: sendCommandOutputOne.Command?.CommandId,
    InstanceId: instanceId,
  },
);

const commandTwoWaiterController = new AbortController();
const commandTwoWaiterPromise = waitUntilCommandExecuted(
  {
    client: ssmClient,
    maxWaitTime: 30,
    abortSignal: commandTwoWaiterController.signal,
  },
  {
    CommandId: sendCommandOutputTwo.Command?.CommandId,
    InstanceId: instanceId,
  },
);

const winningCommandWaiter = await Promise.race([
  commandOneWaiterPromise,
  commandTwoWaiterPromise,
]);

const winningGetCommandInvocationCommandOutput =
    winningCommandWaiter.reason as GetCommandInvocationCommandOutput;
const winningCommandId = winningGetCommandInvocationCommandOutput.CommandId;

assert.equal(
  winningCommandId,
  sendCommandOutputTwo.Command?.CommandId,
  dedent`
    Expected command #2 to finish first. Instead command #1 won.
    Command #1 ID: ${sendCommandOutputTwo.Command?.CommandId}
    Command #2 ID: ${sendCommandOutputOne.Command?.CommandId}
  `,
);

commandOneWaiterController.abort();

await ssmClient.send(
  new CancelCommandCommand({
    CommandId: winningCommandId,
  }),
);

Once command #2 is determined to be the winner, we have to do a bit of clean-up. We abort command #1’s waiter thread, so it stops polling. But aborting the waiter won’t stop the Command Invcation. That’s done by calling CancelCommand.

Read more about waiters in:

• Waiters in modular AWS SDK for JavaScript on the AWS Developer Tools Blog.
• Waiters and signers on the AWS SDK for JavaScript documentation.

Well that shouldn’t stop us. The AWS SDKs actually ship with a really easy way to build waiters ourselves.

In that previous post, I showed you how I used AWS SDK waiters to wait for an SSM Command Invocation to complete. I’d like to do something similar for an SSM Automation. But SSM doesn’t (currently) ship an official waiter for automations. So I wrote my own.

/**
 * @constructor is private to prevent instantiation from outside the Builder
 * @property ssmClient the real SSM client. 
 *   This waiter implements the full SSM waiter interface through Kotlin delegation.
 */
class ExtendedSsmWaiter private constructor(
  private val ssmClient: SsmClient,
  private val waiterOverrideConfiguration: WaiterOverrideConfiguration
) : SsmWaiter by ssmClient.waiter() {
  /**
   * The waiter function.
   * When implementing waiters, think of them encapsulating a desired state and a way to query that state.
   * The desired state is expressed through the function itself. 
   * The desired state of this function is that an automation is executed.
   * The way to query the state is through the getAutomationExecution call and the [ssmClient] property.
   *
   * @return a waiter response object.
   *   If successful this will contain the [GetAutomationExecutionResponse] object from the last successful call.
   */
  fun waitUntilAutomationExecuted(getAutomationExecutionRequest: GetAutomationExecutionRequest): WaiterResponse<GetAutomationExecutionResponse> {
    val waiter = Waiter.builder(GetAutomationExecutionResponse::class.java)
      /**
       * A waiter builder is looking for two fields: acceptors and override configuration.
       *
       * [WaiterAcceptor]s is a series of predicate functions looking to advance the waiter state.
       * While a waiter is running, it will poll the [getAutomationExecutionRequest] call and pass the response to the acceptors.
       *
       * There are three waiter states that the acceptors can advance to: success, retry, and error.
       * Each type of acceptor can act on a normal response or an exception.
       * Which means that there are (`3 × 2 =`) 6 types of acceptors in total.
       *
       * The AWS SDK makes these available as static methods on the [WaiterAcceptor] interface.
       *
       * Waiters work by making the [getAutomationExecutionRequest] call and then passing the response into the acceptors.
       * They then one-by-one through the acceptors in the order they are defined.
       * If one of the acceptors returns `true`, the waiter will advance to whatever the acceptor tells it to.
       * If the acceptors return `false`, the waiter will continue on to the next acceptor.
       * If the acceptors throw an exception, the waiter will stop and the exception is returned.
       */
      .acceptors(
        listOf<WaiterAcceptor<in GetAutomationExecutionResponse>>(
          /**
           * There is likely only one or two responses that you'd consider successful.
           * This is the one state that you really want your waiter to reach.
           */
          WaiterAcceptor.successOnResponseAcceptor {
            it.automationExecution().automationExecutionStatus() == AutomationExecutionStatus.SUCCESS
          },
          /**
           * There are likely quite a few responses that warrant retrying the call.
           * Certainly there are more than what I've listed here.
           * Looking through the [AutomationExecutionStatus] I see twenty unique states there.
           * Maintaining this list here is probably the best argument in favour of always trying to use the AWS-vended waiters when possible.
           */
          WaiterAcceptor.retryOnResponseAcceptor {
            it.automationExecution().automationExecutionStatus() == AutomationExecutionStatus.IN_PROGRESS
          },
          WaiterAcceptor.retryOnResponseAcceptor {
            it.automationExecution().automationExecutionStatus() == AutomationExecutionStatus.PENDING
          },
          /**
           * Never forget to check for failure states.
           * Failure are not successes (duh!) but also mean that there's no point retrying the waiter.
           * They're non-successful terminal states.
           * When you forget failure states, the waiter will just keep retrying until the timeout is reached.
           */
          WaiterAcceptor.errorOnResponseAcceptor(
            {
              it.automationExecution().automationExecutionStatus() == AutomationExecutionStatus.FAILED
            },
            /**
             * Supplying a message on failure is a good idea.
             * The message is used in the exception thrown by the waiter.
             * It's a great way to quickly surface information about the resource that you're waiting on.
             * The more you reveal here, the quicker you'll debug a failure.
             */
            """
              Automation execution has status FAILED. Waiter transitioned to a failure state. 
              Automation Execution: $getAutomationExecutionRequest
            """.trimIndent()
          ),
        )
          /**
           * [WaitersRuntime.DEFAULT_ACCEPTORS] just instruct the waiter to retry on a response.
           * It essentially looks like this:
           *
           * ```kotlin
           * WaiterAcceptor.retryOnResponseAcceptor { true }
           * ```
           *
           * Including it last here means that the waiter will always retry non-exception responses if no other waiter matched earlier.
           */
          .plus(WaitersRuntime.DEFAULT_ACCEPTORS),
      )
      /**
       * Override configuration tells the waiter how long to poll for, which backoff strategy to use and how many attempts to make.
       */
      .overrideConfiguration(waiterOverrideConfiguration)
      .build()

    return waiter.run { ssmClient.getAutomationExecution(getAutomationExecutionRequest) }
  }

  companion object {
    fun builder(): Builder = Builder()
  }

  class Builder internal constructor() {
    private lateinit var ssmClient: SsmClient

    /**
     * Default waiter configurations can be quite tricky to pick.
     * For something like EC2's `waitUntilInstanceRunning`, you can make an educated guess on how long to wait.
     * But other processes may depend on what work you're waiting to happen.
     * [waitUntilAutomationExecuted] is an example of the latter type where the duration depends on what the automation is doing.
     *
     * As an example, EC2's [software.amazon.awssdk.services.ec2.waiters.DefaultEc2Waiter.instanceRunningWaiterConfig] sets a config of:
     * 40 total attempts × 15-second fixed delay backoff = 600 seconds or 10 minutes.
     *
     * In this example I've not set an upper bound on `maxAttempts`.
     * You either need to set max attempts or a wait timeout. Not both.
     * And wait timeout is far easier to reason about.
     * Just ask yourself, "how long should I wait for this to happen?"
     *
     * I've set it to wait 5 seconds between each attempt for a tight feedback loop.
     * Generally longer, wait timeouts can come with longer delays between attempts.
     *
     * I've set a wait timeout of 2 minutes as that's pretty reasonable for my use-case.
     */
    private var waiterOverrideConfiguration: WaiterOverrideConfiguration = WaiterOverrideConfiguration.builder()
      .maxAttempts(Int.MAX_VALUE)
      .backoffStrategyV2(BackoffStrategy.fixedDelay(5.seconds.toJavaDuration()))
      .waitTimeout(2.minutes.toJavaDuration())
      .build()

    fun withWaiterOverrideConfiguration(waiterOverrideConfiguration: WaiterOverrideConfiguration): Builder {
      this.waiterOverrideConfiguration = waiterOverrideConfiguration
      return this
    }

    fun withSsmClient(ssmClient: SsmClient): Builder {
      this.ssmClient = ssmClient
      return this
    }

    fun build(): ExtendedSsmWaiter = ExtendedSsmWaiter(ssmClient, waiterOverrideConfiguration)
  }
}

And then you use it just like you would any other waiter:

val ssmClient = SsmClient.builder().build()
val ssmWaiter = ExtendedSsmWaiter.builder()
  .withSsmClient(ssmClient)
  .build()

val startAutomationResponse = ssmClient.startAutomationExecution(
  StartAutomationExecutionRequest.builder()
    .documentName("AWS-ConfigureCloudWatchOnEC2Instance")
    .parameters(mapOf(
      "InstanceId" to listOf(instanceId)
    ))
    .build()
)

val getAutomationExecutionResponseWaiterResponse = ssmWaiter.waitUntilAutomationExecuted(
  GetAutomationExecutionRequest.builder()
    .automationExecutionId(startAutomationResponse.automationExecutionId())
    .build()
)

assertThat(getAutomationExecutionResponseWaiterResponse.attemptsExecuted())
  .isGreaterThan(0)
assertThat(getAutomationExecutionResponseWaiterResponse.matched().response())
  .isPresent

Hopefully you can also see that there’s nothing AWS-specific about implementing a custom waiter. So write them for: AWS’s resources, other service’s resources, and even your own resources. ✨

2025-10-05: I’ve written about using waiters in Javascript/Typescript. Read about it on Typescript waiters are a bit weird.

2025-09-27: Continuing on with waiters, I’ve written a new post about building your own waiters for an resource. Read more at Write custom waiters.

Most distributed systems are eventually consistent. Callers instruct a resource to change from one state to another (like going from running to stopped) and must then wait some amount of time before that new state is achieved.

Writing your own polling code to manage this adds complexity, maintenance cost and the opportunity for bugs. It’ll either be too simple and not cover all of the edge cases (what if instead of going to stopped, it goes into failed) or too complex and force the user to consider exponential back-off and jitter.

When used correctly, AWS SDK waiters can be the perfect tool to help one system drive consistency in another. They’re a pattern that helps to make your code simpler, easier to reason about, and more resilient.

An example

Since waiters wait for a resource to reach a certain state, and I want to provide real examples in this post, I’ll need a resource to use. From here on I’ve used an AWS SSM Command Invocation (archive) as the resource. There’s nothing particularly special about SSM or a Command Invocation that makes it more suitable as a waiter, it’s just a resource that is eventually consistent.

If you’re not already familiar, SSM SendCommand executes an SSM document on a target EC2 instance. An SSM document is usually a series of shell scripts. It can take some time after you call SendCommand for the command to reach the instance and it can take some time after that for the document to run; depending on what the document does.

I’ve also used AssertJ assertions throughout the following examples. Waiters are particularly useful during integration and canary testing and AssertJ provides a nice way to show the reader the expected behaviour. Read more about why I use an assertion library and why I choose AssertJ over others elsewhere in this blog.

Here how to a waiter to wait until an SSM Command Invocation completes. The command itself is super basic. It just echo’s Hello, World! before completing.

val ssmClient = SsmClient.builder().build()
val ssmWaiter = ssmClient.waiter()

val sendCommandResponse = ssmClient.sendCommand(
  SendCommandRequest.builder()
    .instanceIds(listOf(instanceId))
    .documentName("AWS-RunShellScript")
    .parameters(mapOf(
      "commands" to listOf(
        /* https://kieran.casa/starting-bash-scripts/ */
        "#!/usr/bin/env bash",
        "set -o xtrace",
        "set -o errexit",
        "set -o nounset",
        "set -o pipefail",
        "echo 'Hello, World!'",
        "exit 0"
      )
    ))
    .build()
)

val getCommandInvocationResponseWaiterResponse = ssmWaiter.waitUntilCommandExecuted(
  GetCommandInvocationRequest.builder()
    .commandId(sendCommandResponse.command().commandId())
    .instanceId(sendCommandResponse.command().instanceIds().first())
    .build()
)

assertThat(getCommandInvocationResponseWaiterResponse.attemptsExecuted())
  .isGreaterThan(0)
assertThat(getCommandInvocationResponseWaiterResponse.matched().response().get().standardOutputContent())
  .contains("Hello, World!")

Execution of your code will pause at ssmWaiter.waitUntilCommandExecuted until one of the following happens:

• The command invocation successfully completes. As shown above.
• The command ends in a failure
• The waiter times out

Failure modes

When a command ends in failure, expect the waiter to raise an exception:

val sendCommandResponse = ssmClient.sendCommand(
  SendCommandRequest.builder()
    .instanceIds(listOf(instanceId))
    .documentName("AWS-RunShellScript")
    .parameters(mapOf(
      "commands" to listOf(
        "#!/usr/bin/env bash",
        "exit 1"
      )
    ))
    .build()
)

assertThatThrownBy {
  ssmWaiter.waitUntilCommandExecuted(
    GetCommandInvocationRequest.builder()
      .commandId(sendCommandResponse.command().commandId())
      .instanceId(sendCommandResponse.command().instanceIds().first())
      .build()
  )
}
  .isInstanceOf(SdkClientException::class.java)
  .hasMessageContaining("A waiter acceptor with the matcher (path) was matched on parameter (Status=Failed) and transitioned the waiter to failure state")

And the same goes for when a waiter times out:

val sendCommandResponse = ssmClient.sendCommand(
  SendCommandRequest.builder()
    .instanceIds(listOf(instanceId))
    .documentName("AWS-RunShellScript")
    .parameters(mapOf(
      "commands" to listOf(
        "#!/usr/bin/env bash",
        "sleep 20",
        "exit 0"
      )
    ))
    .build()
)

assertThatThrownBy {
  ssmWaiter.waitUntilCommandExecuted(
    GetCommandInvocationRequest.builder()
      .commandId(sendCommandResponse.command().commandId())
      .instanceId(sendCommandResponse.command().instanceIds().first())
      .build()
  )
}
  .isInstanceOf(SdkClientException::class.java)
  .hasMessageContaining("The waiter has exceeded the max wait time or the next retry will exceed the max wait time + PT5S")

And when a waiter exceeds the maximum number of attempts:

assertThatThrownBy {
  ssmWaiter.waitUntilCommandExecuted(
    GetCommandInvocationRequest.builder()
      .commandId(sendCommandResponse.command().commandId())
      .instanceId(sendCommandResponse.command().instanceIds().first())
      .build()
  )
}
  .isInstanceOf(SdkClientException::class.java)
  .hasMessageContaining("The waiter has exceeded the max retry attempts: 1")

Tuning timeouts

AWS SDK waiters come with default wait times, max attempts and backoff strategies. Much the same as what you’d expect to see when configuring retries. However, depending on the resource, different wait times may be more acceptable than the default.

In the SSM waiter example, you really want to set your wait time to whatever is appropriate for the command that you’re running. If the command runs for about an hour, then the waiter should be ready to wait about that long. They’re configurable like this:

val ssmWaiter = SsmWaiter.builder()
  .client(ssmClient)
  .overrideConfiguration(
    WaiterOverrideConfiguration.builder()
      .maxAttempts(Int.MAX_VALUE)
      .waitTimeout(60.minutes.toJavaDuration())
      .backoffStrategyV2(BackoffStrategy.fixedDelay(5.seconds.toJavaDuration()))
      .build()
  )
  .build()

Read more about waiters at Using waiters in the AWS SDK for Java 2.x (archive). Waiters are also available in other languages (archive).

I see this everywhere:

const allItems = [];
let nextToken: string | undefined;

do {
  const response = await dynamodb.scan({
    TableName: "MyTable",
    ExclusiveStartKey: nextToken,
  });

  allItems.push(...(response.Items || []));
  nextToken = response.LastEvaluatedKey;
} while (nextToken);

But this approach has some problems:

• There’s mutable state with allItems and nextToken. Looking at you, let.
• It’s easy to introduce bugs with loop conditions. You’ve got to make sure you’re doing a do while loop, not a while loop.
• There can be memory issues with large result sets. This solution doesn’t give you the chance to process just a single page of results.
• You have to be aware of the pagination token, and then handle it correctly. For example, with DDB, you must remember that the LastEvaluatedKey becomes the ExclusiveStartKey.

To solve most of these problems, the AWS SDK ships built-in paginators for all paginated operations:

import { paginateScan } from "@aws-sdk/lib-dynamodb";

const allItems = [];
for await (const page of paginateScan(
  /* DynamoDBPaginationConfiguration */ { client: dynamodb },
  /* ScanCommandInput */ { TableName: "MyTable" }
)) {
  allItems.push(...(page.Items || []));
}

Their benefits are:

• Eliminates most mutable state. No more let variables for pagination tokens.
• Automatic pagination tokens management. The SDK handles pagination tokens automatically.
• Simpler loop logic. Just iterate until finished, no do while complexity.

Paginators are not just for JS/TS either. There’s paginator support in:

• Java (archive)
• Python (archive)
• Go (archive)

And I’m sure others, too.

Some more examples

CloudWatch Logs:

import { paginateDescribeLogGroups } from "@aws-sdk/client-cloudwatch-logs";

for await (const page of paginateDescribeLogGroups(
  { client: cloudwatchLogs },
  {}
)) {
  console.log(page.logGroups);
}

EC2 Images:

import { paginateDescribeImages } from "@aws-sdk/client-ec2";

for await (const page of paginateDescribeImages(
  { client: ec2 },
  { Owners: ["self"] }
)) {
  console.log(page.Images);
}

Stop writing manual pagination loops. Use the SDK’s paginators instead.

A Utility Function

For cases where you need all results in an array, this utility function is helpful:

export const toArray = async <T>(gen: AsyncIterable<T>): Promise<T[]> => {
  const out: T[] = [];
  for await (const x of gen) {
    out.push(x);
  }
  return out;
};

// Use it like this

const pages = await toArray(
  paginateScan({ client: dynamodb }, { TableName: "MyTable" })
);
const allItems = pages.flatMap((page) => page.Items || []);

You’ll notice that we got rid of allItems from above! It is now hidden in the toArray function. The mutability is constrained to the scope of the toArray function.

Links

• Pagination using Async Iterators in modular AWS SDK for JavaScript | AWS Developer Tools Blog (archive)
• paginateScan for the AWS SDK for JavaScript v3 (archive)
• paginateDescribeLogGroups for the AWS SDK for JavaScript v3 (archive)
• paginateDescribeImages for the AWS SDK for JavaScript v3 (archive)