Developer Center

Resources to get you started with Algorithmia

Event Listeners via Azure SB

Updated

These directions will help you to set up an Algorithmia Event Listener, which will run an algorithm with input payloads provided in an Azure Service Bus Queue. [Read More about Azure Service Bus] (https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues)

This is only available for Enterprise installations of Algorithmia.

1. Obtain a template file and account info from Algorithmia

Contact support@algorithmia.com to obtain the following, which you will need during CloudFormation setup:

  • Azure Resource Manager(ARM) template for Service Bus Namespace and Queue
  • Custom Role definition file - QueueReceiver
  • Algorithmia Azure Account

2. Invite Algorithmia account as guest account in your account’s Active Directory

Using the Custom Role definition file you received in step 1, create the Custom Role in your account.

  • Authenticate with Azure CLI
  • Run $ az role definition create --role-definition GuestRole.json

More info on how to do this with Azure CLI can be found here: https://docs.microsoft.com/en-us/azure/role-based-access-control/custom-roles-cli It can be also done via PowerShell and REST API.

3. Invite Algorithmia account as guest account in your account’s Active Directory

Create a new guest user account with Algorithmia’s account email.

  • Sign in to the Azure portal as an Azure AD administrator.
  • In the left pane, select Azure Active Directory.
  • Select New guest user.
  • On the New user page, select Invite user and then add the guest user’s information.
    • Name. The first and last name of the guest user.
    • Email address. Use the email obtained in step 1
    • Groups: You can add the guest user to one or more existing groups, or you can do it later.
    • Directory role: Use the role you created in step 2 (QueueReceiver)
    • Select Invite to automatically send the invitation to the guest user. A notification appears in the upper right with the message Successfully invited user. The user account is now added to the directory as a guest account with the custom role.

4. Create Service Bus Namespace and Queue

  • Import ARM template and create your Service Bus Namespace and Queue. Info about ARM templates: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/overview Info about Service Bus: https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-quickstart-portal#:~:text=In%20the%20left%20navigation%20pane,if%20the%20name%20is%20available
  • Create a new Template, and load the contents of the ARM template file from step 1 into the ARM template tab. Save the template and click “Deploy Template”.
  • Fill in the Namespace Name, Queue Name, and choose region appropriate for you.

Note: If the deployment fails, chances are there is already a namespace created with that name in the region you chose. If that is the error message you get on deployment error, you would need to redeploy with a different name until it works.

5. Test Event Listener

  • Create a new Python 3.x Algorithm.

  • In the web IDE replace the source with the following:

import Algorithmia

def apply(input):
    Algorithmia.client().file("data://<username>/event_output_directory/" + input.get("filename")).put(input.get("data"))

  • Click Save, then Build, then Publish buttons. On the algorithm page copy the full “path” of the algorithm.
  • Create Hosted Data directory “event_output_directory”
  • Create event listener -> Azure, algorithm name, algorithm version
  • To test event listener, create a new Python 3.x algorithm just like before, and replace the source code with the following:
import Algorithmia
from azure.servicebus import QueueClient, Message

def apply(input):
    # Create the QueueClient
    queue_client = QueueClient.from_connection_string("<Connection String>", "<Queue Name>")

    # Send a test message to the queue
    msg = Message(b'{"filename":"test","data":"test-data"}')
    queue_client.send(msg)    
  • Add these dependencies to the created algorithm:
azure-common==1.1.25   
azure-servicebus==0.50.2

  • Run the algorithm that would send messages
  • Check the event_output_directory for a file with filename “test”
  • Download the file, it should contain text “test-data”, your event listener is working now!