Meet Carrier: A lightweight messaging adapter for webhooks

At Amplify, we’ve discussed how open source is an integral part of our culture in our previous engineering blog. In that spirit, we’re excited to open source our SQS messaging adapter for webhooks, Carrier. Carrier is also released as a public Docker image on Docker Hub.

Bottom Line Up Front

Carrier is a messaging adapter for webhooks which helps developers implement the AWS Elastic Beanstalk worker environment (a common event worker pattern) outside of the Elastic Beanstalk service. This allows for portable workers that can easily be deployed anywhere, like Kubernetes, by exposing a webhook and deploying Carrier alongside them. Carrier supports dynamic SQS visibility timeouts and dynamic message body MIME types. Additionally, carrier is lightweight, with an image size of only 8MB and consuming only 43MB of RAM under medium workloads. Like Probe, Carrier is made public under the MIT license.

Event Driven Architecture with SQS

Event workers are a common pattern in Event Driven Architecture in which processes asynchronously respond to events generated by other external or internal systems. Amazon Simple Queue Service (SQS) is a popular messaging technology commonly utilized in Event Driven Architectures. AWS SQS can be utilized as a queue for messages originally generated by an AWS Simple Notification Service (SNS) topic or events originating from an AWS S3 bucket. The number of event workers processing messages from any particular SQS queue can easily scale up or down in response to the number of messages available on the queue. For these reasons, utilizing SQS queues to facilitate the event worker pattern is a battle tested and flexible strategy for deploying an Event Driven Architecture. One question, however, remains: what’s the best way to consume messages from an SQS queue? At Amplify, we believe the best way to consume messages from SQS is through the “worker” pattern originally developed by AWS in the Elastic Beanstalk service.

Event Workers with Webhooks

The Amazon Elastic Beanstalk worker pattern has a worker service which exposes a webhook endpoint and an SQS daemon, or SQSd, which consumes the messages from the SQS queue and forwards the messages to the webhook. In our experience, this pattern works well because it enables developers to build workers using tested HTTP servers that they already know (like gin, uvicorn, or rocket) instead of custom event loops. There are many other benefits to the worker pattern:

• Improved single-process scaling, as most modern HTTP servers already support concurrency.
• Improved monitoring and logging for metrics like response codes and latency via familiar HTTP server support.
• Loosely coupled workers allow for incredible technology flexibility. We’ll discuss this more below.

Carrier is a replacement for the SQS daemon in the worker pattern.

Why We Built Carrier

There are many existing SQS daemon replacements like mozart-analytics/sqsd, mogadanez/sqsd, and olxbr/aws-sqsd and we’ve used them before at Amplify! However, there were a few ways we felt that we could improve on the basic formula for an SQS daemon replacement. First, most of these projects do not publish easy to consume Docker images with the exception of mogadanez/sqsd. In fact, if you search “sqsd” on Docker Hub, you will find most images are simply forks of the mozart-analytics/sqsd project and published on Docker Hub. Fun fact: the socialware/sqsd image was one of the first public images I published on Docker Hub! Second, existing SQS replacements available on Docker Hub utilize heavy components like Node.js or the JVM, which leads to large image sizes. For example, the mogadanez/sqsd image is 987MB . Finally, these heavy components like Node.js or the JVM also contribute to high memory utilization. For an image that gets deployed alongside every single worker, the memory utilization can really add up across an entire deployment! Amplify set out to build Carrier not only to solve these issues but also to implement specific functionality necessary for our business.

Dynamic Visibility Timeouts

At Amplify, we have to work with a lot of LLM calls. These new APIs still offer unreliable latency at best and frequently throttle. While exponential backoffs and other retry patterns are common within a single process, it’s not a distributed pattern to limit usage of an external API or other limited system. It’s entirely possible to burn through multiple retries and multiple receives for a single message before an external service like an LLM API is available again or token limits are reset. This leads to messages ending up in the Dead Letter Queue, which isn’t an optimal solution and requires redriving the DLQ.

SQS visibility timeouts, on the other hand, can be used to implement backoff patterns across an entire distributed service and with much longer backoff times. The maximum SQS visibility timeout is 12 hours! Utilizing visibility timeouts correctly means that your service can attempt to reprocess a message much later, hopefully when token limits have been reset or APIs are performing more predictably. Carrier supports the HTTP 429 Too Many Requests response code along with the Retry-After header to set visibility timeouts on messages. Carrier also sends the X-Carrier-Receive-Count and X-Carrier-First-Receive-Time headers with each POST to the webhook. These headers allow the webhook logic to understand how many times the message has been received and when it was first received to calculate traditional exponential backoff schemes when necessary. The backoff period can then be communicated to Carrier by returning a 429 Too Many Requests response code with the Retry-After header set to the Unix timestamp of when the next retry should occur. Carrier will do the calculation of converting the Unix timestamp to a new visibility timeout and communicate that to SQS.

Dynamic Message Body MIME Types

Carrier also supports dynamic message body MIME types utilizing an SQS message attribute. Carrier will read any SQS message attribute of Type String with Name Body.ContentType and forward this value in the Content-Type header. You can send a message to SQS with a Body.ContentType message attribute by utilizing the AWS SDK for your preferred language. For example, in Go:

package main

import (
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/sqs"
)

func main() {
	cfg, _ := config.LoadDefaultConfig(context.Background())
	client := sqs.New(cfg)
	r := client.SendMessageRequest(context.Background(), &sqs.SendMessageInput{
		DelaySeconds:      0,
		MessageAttributes: map[string]types.MessageAttributeValue{
			"Body.ContentType": {
				DataType:    aws.String("String"),
				StringValue: aws.String("application/yaml"),
			},
		},
		MessageBody:       aws.String("---\ncontent: \"This is YAML\"\n"),
		QueueUrl:          aws.String("sqs.us-west-2.amazonaws.com/1234567890/carrier-demo"),
	}
	_, _ := r.Send()
}

The default Content-Type header is always application/json if unspecified. Carrier also supports setting a static, but non-default, MIME type for the Content-Type header with the CARRIER_WEBHOOK_DEFAULT_CONTENT_TYPE environment variable.

More than just SQS?

One of the great strengths of the worker pattern we’ve been discussing is that it loosely couples the business logic of the event handler with the underlying message technology. The worker exposing the webhook doesn’t even need to know that it’s consuming messages from SQS! To take full advantage of this strength of the architecture pattern, Carrier was designed from the start with the idea that it can, and should, support more messaging technologies than just SQS and adapt them to a well-defined webhook API. If you build a worker that supports Carrier, you should be able to deploy that worker across platforms and technologies. While Carrier currently only supports SQS, the basic architecture to support other messaging technologies, like Azure Queue Storage, the numerous *MQ platforms like RabbitMQ and ElasticMQ, and even streaming technologies, like Kafka and AWS Kinesis, is already in place. We are excited to see what Carrier can become and how flexible it can be!

One More Thing…

Image size and performance were crucial to us while building Carrier. As a special callout, we’d like to share that the Carrier image is over 126X smaller than currently popular sqsd images (987MB vs 7.8MB) and uses 8.6X less RAM at idle (43MB vs 5MB). The RAM savings continue under load but your mileage may vary depending on your workload. We’ve also tried to keep 3rd party dependencies to a minimum to help keep your supply chain secure. Finally, we’ve made Carrier public under the MIT license so that you can incorporate it into your environment without licensing concerns. Finally, Carrier does not support the older XML API used by previous versions of SQS and ElasticMQ. As such, useful tools for local development like roribio16/alpine-sqs will not work with Carrier without updates. We have opened a PR to the roribio/alpine-sqs GitHub repository to update ElasticMQ to version 1.6.1 which supports the latest JSON protocol. However, until this PR is merged, we are also publishing the updated version of the image at amplifysecurity/alpine-sqs. Feel free to use this image to test Carrier locally.

Wrap Up

We hope you find Carrier useful in your deployment environment. Amplify remains committed to supporting our open source projects and would love to hear about any issues you might find while using Carrier. If you like to see open source projects like Carrier, please give us a star and support our open source efforts! Currently, the latest Carrier image is 0.1.1 and some API changes may occur as we continue towards 1.0.0. Full examples of how to deploy Carrier using Kubernetes are available in the README both in the Carrier GitHub Repository and Docker Hub image registry.