Developer Center

Resources to get you started with Algorithmia

Amazon SQS Message Broker


These directions will help you to set up an Algorithmia Event Listener, which will run an algorithm with input payloads provided in an Amazon Simple Queue Service (SQS) Queue.

This feature is only available for Enterprise installations of Algorithmia.

SQS Event Listener workflow

SQS and Algorithmia Workflow Diagram

This shows the interaction of SQS and Algorithmia. After consuming a message from SQS, the Algorithmia Event Listener will send it as an API call to Algorithmia. If API server returns a 200 OK then the message was queued for processing. If the API server returns 429 Too Many Requests, then the message is put back into the SQS queue with a 5 minute delay.

To affect how much work the system will accept, you can tune either the maximum number of workers, or the user session limit. The SQS option for “Maximum Receives” determines how many times a message will loop through this system. Since each loop is 5 minutes, messages will queue on the SQS side for (5 minutes) * (Max Receives). As per the AWS documentation, the default Max Receives is 10 but can be increased up to 1000 to queue messages for longer. When Max Receives limit is hit, the message is delivered to the SQS Dead Letter Queue.

1. Obtain a template file and account info from Algorithmia

Contact to obtain the following, which you will need during CloudFormation setup:

  • the CloudFormation template file, client-aws-cloudformation-template.yaml
  • AlgorithmiaAccountNumber
  • QueueName
  • QueueDLQName

2. Set AWS user permissions

You will need to have a user (or role that your user can assume) with the following AWS permissions:

Cloud Formation: CreateStack and DescribeStacks (on all resources)

IAM: CreateRole, GetRole, and PutRolePolicy (on all resources)

SQS: CreateQueue, GetQueueAttributes, and SetQueueAttributes (on all resources)

3. Creating Event Listener resources in AWS

Open the CloudFormation page in the AWS console.

Click the Create stack button.

On the following page, select Upload a template file, then click Choose file. Navigate to the Event Listener template file provided by Algorithmia and select it.

Click Next.

On the following page, enter the Stack name. This should be a unique name that identifies the package of Amazon resources used for your event listener.

Enter the AlgorithmiaAccountNumber. This is Algorithmia’s AWS account number, provided to you by Algorithmia.

Enter the QueueName. This should be a unique name that identifies the Amazon SQS queue that will be the source of payloads for the algorithm to run.

Enter the QueueDLQName. This should be a unique name that identifies the Amazon SQS queue that will be a holding queue for payloads that were not accepted by the algorithmia algorithm for any reason (invalid message format, Algorithmia platform downtime, etc)

Click Next.

The next page allows you to Configure stack options. If you need to use a specific IAM role which has the permissions specified in section 1), enter that role in the Permissions section, otherwise, leave it blank.

Click Next.

The next page allows you to review your options. Scroll to the bottom and click the checkbox next to I acknowledge that AWS CloudFormation might create IAM resources.

Finally, click Create stack.

The next page will indicate that the stack creation is in progress.

Click the Stack info tab.

After about 60 seconds, click the grey refresh wheel () . It should show the Stack status as CREATE_COMPLETE . If not, wait another 60 seconds and click the wheel again.

Click the Outputs tab. Copy the QueueURL and QueueConsumerARN; you will use them when creating your Event Listener in the Algorithmia web user interface next.

4. Creating an Event Listener in the Algorithmia platform

Any algorithm can be used with an Event Listener. This tutorial shows one possible sample algorithm.

Open the Algorithmia Web user interface and log in.

Click the Create New button and select Algorithm.

Scroll down and select “Python 3.x”

Scroll down and click Create New Algorithm.

On the following page, click the Source tab for the newly created algorithm.

Replace the original source with the following:

import Algorithmia

def apply(input):
    Algorithmia.client().file("data://traack/event_output_directory/" + input.get("filename")).put(input.get("data"))

Click the Save, the Build, then Publish buttons.

On the Algorithm page, copy the full “path” to the algorithm, in this case, traack/EventListenerAlgo/0.1.1.

Click the Data Sources button in the left sidebar.

Click My Hosted Data on the main screen.

Click the New Collection button in the top right corner.

In the dialog box that appears, enter event_output_directory as the collection name, then click Create Collection.

Click the Home button the left sidebar.

Click on the Create New button and select Event Listener.

Enter the QueueURL and QueueConsumerARN into the URI and ROLE ARN fields on the following page, then scroll down.

Enter the full path of your algorithm published earlier and click Create New Event Listener. The following page will show your Event Listener configuration.

5. Triggering the Event Listener

Many methods exist for sending messages to AWS SQS queues. Here is one method using the AWS SQS queue web user interface.

Open the SQS page in the AWS console. Click on the box to the left of the queue name that was created in step 2.

Click the Queue Actions menu, and click Send a Message.

In the popup box, enter the message payload corresponding to the Algorithm inputs to which the Event Listener is linked:

Click Send Message.

At this point, the message should be immediately consumed by the Event Listener and sent to the algorithm for execution.

Open the Algorithmia Data API page for the directory to be written to in your algorithm.

Note that the directory contains a file with the filename specified in the payload and that the contents of the file include the data from the payload.

If you’re new to Algorithmia and would like to learn more about our product and using event listeners with Amazon SQS, please contact our sales team. We’d love to hear from you!