In this tutorial, learn how you can monitor your model performance metrics with Datadog using the new Insights feature of Algorithmia Enterprise.

Algorithmia integration: How to monitor model performance metrics with Datadog

Algorithmia Insights is a new feature of Algorithmia Enterprise and provides a metrics pipeline that can be used to instrument, measure, and monitor your machine learning models. Monitoring your model performance metrics can help with your organization’s overall AI/ML governance efforts, and use cases include detecting model drift, data drift, and model bias.

Algorithmia Datadog integration

Datadog is a cloud-based monitoring service that aggregates metrics and logs across different services. You can use Insights to send real-time model performance metrics from Algorithmia to Datadog as time-series data that can be included in dashboards, monitors, and alerts.

Animation showing an overview of the Algorithmia Datadog integration

The Algorithmia-Datadog integration described in this post allows you to stream operational and inference metrics from your models in Algorithmia to Datadog, where you can create dashboards, monitors, and alerts to analyze model performance metrics in production systems across your entire model catalog.

How the Algorithmia-Datadog integration works

When algorithms that have Insights enabled are queried, Algorithmia emits a metrics payload to a configured Kafka broker and topic. The Algorithmia-Datadog integration then transforms and forwards Insights data from Kafka to the Datadog metrics API endpoint. All of the data from Algorithmia Insights will show up in Datadog as metrics with the prefix algorithmia. This makes it easy to include model performance data from Algorithmia Insights in dashboards or monitors.

Diagram showing the Algorithmia Datadog integration

The metrics payload from Algorithmia Insights contains both operational and inference metrics. Operational metrics include the algorithm name, version, owner, duration, session ID, and request ID. Inference metrics include user-defined metrics that are specified by data scientists depending on the particular algorithm and use case.

Diagram showing the metrics payload

When this integration is used, all of the operational metrics related to the algorithm name, version, owner, request ID, and session ID will be included with each data point as tags in Datadog. These tags can then be used to filter and group model performance metrics in dashboards and monitors.

The steps to install and configure this integration can be summarized as:

  1. Configure Algorithmia to send Insights to a Kafka broker and topic

  2. Install and run the Algorithmia-Datadog integration with your Kafka broker/topic and Datadog API key

  3. From Algorithmia, you can then instrument, publish, and query algorithms that have Insights enabled

  4. Verify that your metrics appear in Datadog as algorithmia.<METRIC-NAME>

Once you’ve installed this integration in Datadog, you’ll have a default dashboard and monitor to help you get started with Insights and Datadog:

Default dashboard and monitor

Refer to the integration page on Datadog and documentation in the Algorithmia integrations repository for more detailed information on installing and configuring this integration.

Example #1: Creating a dashboard with model performance metrics

Datadog provides dashboards for visually tracking, analyzing, and displaying key performance metrics, which enable you to monitor the performance of your models in real time.

In this example, we’ll walk through the steps to create a new dashboard in Datadog and visualize metrics from Algorithmia Insights.

From the Dashboard page in Datadog, click on the New Dashboard button:

Picture of clicking on the new dashboard button

Specify a name for your new dashboard, then click on the New Timeboard button:

Picture of clicking on the New Timeboard button

When viewing the new dashboard, click on the Edit Widgets button. Drag a new widget such as the Timeseries widget onto your dashboard:

Picture of dragging the Timeseries widget onto your dashboard

Once you drag a widget onto the dashboard, a configuration screen will appear where you can select the type of visualization you want to use and which data to show on your graph.

In the Graph your data configuration section, select the metric that you want to display from the list of metrics that start with the prefix algorithmia. For example, we’ll choose the custom algorithmia.risk_score metric that we defined in a model (or you could choose the default operational metric named algorithmia.duration_milliseconds):

Picture of selecting a metric

Click on the Save button, and you should see the data from Insights show up in your new widget. (You might need to query your model in Algorithmia to see data flowing into Datadog.)

You can continue adding different widgets and customizing your dashboard to display different metrics from Insights:

Picture of customizing your dashboard

Now you’ll have all of the relevant model metrics from Insights displayed in one place with real-time updates as models are queried in Algorithmia.

Example #2: Creating a monitor for model performance metrics

Datadog provides monitors that allow you to see when critical changes are occurring by actively checking metrics and more.

In this example, we’ll walk through the steps to create a new monitor in Datadog to alert on metrics from Algorithmia Insights.

From the Monitors page in Datadog, click on the New Monitor button:

Picture of clicking the New Monitor button

Select Metric as the monitor type:

Picture of selecting Metric as the monitor type

In the Define the metric configuration section, select the metric that you want to display from the list of metrics that start with the prefix algorithmia. For example, we’ll choose the custom algorithmia.risk_score metric that we defined in a model (or you could choose the default operational metric named algorithmia.duration_milliseconds):

Picture of the Define the metric configuration section

Set an alert threshold, monitor name, monitor message, notification settings, and any other desired configuration settings. Then, click on the Save button at the bottom of the monitor configuration page:

Picture of setting the configuration settings

If your metrics from Insights exceed the specified threshold value, you’ll receive an alert that includes guidance on what actions need to be taken. Depending on your use case, you might want to configure notifications for email, Slack, PagerDuty, or other notification systems, and your suggested actions might include retraining and publishing a new version of a model, rolling back to an older version of a model, or other actions to mitigate model risk.

Example #3: Comparing performance metrics between model versions

The Algorithmia-Datadog integration sends operational metrics as tags, which can be used to filter and group model performance metrics in dashboards and monitors. Operational metrics include the algorithm name, version, owner, session ID, and request ID.

In this example, we’ll demonstrate how you can use tags in Datadog to compare metrics between model versions.

Create a new dashboard by following the steps in the earlier section to create a dashboard (or, you can edit an existing dashboard). Add a new widget (or edit an existing widget) and navigate to the Graph your data configuration section.

Using the Graph additional option, choose Metrics, then select the same metric twice from an algorithm that has data from at least two versions that you want to compare:

Picture of the Graph your data configuration section

Using the from field for each metric, select two different model versions that you want to compare by filtering for algorithm_version:<X.Y.Z>, where X.Y.Z corresponds to your algorithm versions:

Picture of selecting model versions

Now when you view your dashboard, you should see different series in the graph for each version that you specified in the previous step:

Picture of viewing your dashboard

You can also set monitors and alerts for calculated metrics that measure differences between versions. For example, the following monitor is configured with a threshold alert that will be triggered when the difference between two model versions exceeds a specified value:

Picture of calculated metrics

You can use the same approach with tags to filter metrics by version, name, or owner and compare different metrics from Insights in your dashboards and monitors.

Get started with Algorithmia Insights

Algorithmia Insights makes it easy to monitor and alert on model performance metrics by integrating with popular observability platforms such as Datadog. The examples and use cases described in this post demonstrate how you can develop dashboards and alerts to monitor model performance metrics in production systems across your entire model catalog.

You can use this integration and the examples described here to create initial dashboards and alerts, then implement additional monitoring functionality such as anomaly detection, heat maps, change alerts, forecast alerts, and other capabilities from Datadog. Get started with the Algorithmia-Datadog integration by visiting the integration page on Datadog and repository on GitHub and following the documented steps.

Want to see it in action? Watch our on-demand webinar now for a step-by-step walk-through of Algorithmia Insights and demonstration of how to get started today.

Kristopher Overholt