Developer Center

Resources to get you started with Algorithmia

Algorithmia Insights

Updated

Algorithmia Insights is an algorithm metrics pipeline providing you access to your algorithm metrics payload for each algorithm session so you can measure, monitor, and manage your algorithms in production.

With Algorithmia Insights you’ll be able to opt-in to exporting your algorithm’s operational metrics to external monitoring and alerting tools. This means you’ll be able to monitor your algorithm’s execution time as well as capture any model inference metrics you expose from your algorithm’s output such as predictions or accuracy. Any metrics you opt to collect will be captured and exported in a payload and, with help from a Platform Administrator, can be connected to your external systems for monitoring and alerting.

Algorithmia Insights currently supports Python 3.x and R algorithms and Apache Kafka as a destination for metrics.

Types of Algorithmia Insights metrics

Algorithmia Insights metrics fall into two categories:

  • Operational metrics
  • Inference-related metrics

Operational metrics include (but are not limited to) execution time, request ID, algorithm owner, algorithm name, algorithm version, timestamp, etc. When Algorithmia Insights are turned on for a particular algorithm version, these are automatically collected and reported without any additional action necessary on the part of the algorithm author or manager.

Inference-related metrics are defined by the algorithm author in algorithm code. Each metric is defined by a title, like cute_cats_detected, and a numeric value, like 24. Multiple inference-related metrics can be reported at one time, and the set of metrics reported does not need to be the same from one algorithm execution to another.

One-time administrator setup

The first step in allowing Algorithmia Insights to be collected and reported for algorithms is to have the Algorithmia Platform Administrator connect your cluster to a Kafka broker / cluster to which metrics will be sent.

The Algorithmia platform supports two Kafka connection configurations:

  • SCRAM (Salted Challenge Response Authentication Mechanism) over TLS. Provides encrypted communications, authentication credential database breach protection, man-in-the-middle attack protection, and internationalization support. Please see the Kafka SCRAM setup guide for configuring a Kafka broker / cluster to support this configuration.
  • Unencrypted / plaintext. Provides all of the features of Algorithmia Insights for use without encryption or authentication safeguards; allows for testing / prototyping as well as use in networks with existing robust protection against attacks.

The first step to configuring your cluster for Algorithmia Insights is to navigate to the “Algorithmia Insights” administration page, which can be found under the “System Actions” section.

From here, enter the information about your Kafka broker. The following information is needed for any connection type:

  • Kafka URL: a list of comma-separated Kafka bootstrap servers used to establish the initial connection to the Kafka cluster. Algorithmia will make use of all servers irrespective of which servers are specified here for bootstrapping - this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form host1:port1,host2:port2,... Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).
  • Topic Name: the name of the Kafka topic to which Algorithmia Insights will be published

For each execution of algorithms for which Algorithmia Insights is turned on, the Algorithmia Platform will perform multiple data send attempts to the Kafka broker / cluster over the course of four minutes. If, after that time, the data cannot be sent to the Kafka broker / cluster for any reason (Kafka server offline, failed authenticated, network disconnections, etc), the data for that specific execution will be permanently lost.

Unencrypted / plaintext

Unencrypted, plaintext connections do not require any additional configuration information.

Image of Kafka connection UI for unencrypted, plaintext connections

SCRAM / TLS

Encrypted, SCRAM-authenticated connections also require:

  • Username - the username used to authenticate to the Kafka cluster
  • Password - the password used to authenticate to the Kafka cluster
  • CA Certificate - the Certificate Authority certificate used to sign the TLS certificates that the Kafka servers use for communication.

Image of Kafka connection UI for encrypted, SCRAM connections

Using Algorithmia Insights in an algorithm

Each algorithm in your Algorithmia cluster has independent settings for whether to collect and report Algorithmia Insights. Additionally, each published version of your algorithm can have Algorithmia Insights turned on or off.

Algorithmia client libraries are used to report inference-related metrics that you create inside of your algorithm code. Please see the “Publishing Algorithmia Insights” section of the appropriate client library guide for the language that you’re using.

Restrictions

You can report up to 25 metrics in your code. The Algorithmia Platform will report additional metrics which do not count towards these 25.

Additionally, the total size of your metrics data (keys and values) must not exceed 25 kilobytes.

Finally, there are several reserved metric names that must not be used by client code, as they will conflict with automatically-created metrics for each algorithm execution:

  • request_id
  • timestamp
  • execution_time_ms
  • algorithm_owner
  • algorithm_name
  • algorithm_version

Enabling Algorithmia Insights for an algorithm version

Enabling Algorithmia Insights for an algorithm version happens during algorithm publishing.

When using the web UI publishing workflow, a toggle will appear to enable Algorithmia Insights for the specific algorithm version that you are publishing.

Image of algorithm publish UI with Algorithmia Insights checkbox

When using the HTTP API or client libraries to publish your algorithm, passing the key value pair "insights_enabled":true in the settings hash will enable Algorithmia Insights for your algorithm. For example, when using cURL:

curl -H 'Authorization: Simple YOUR_API_KEY' https://<algorithmia-cluster-host>/v1/algorithms/<algorithm-owner>/<algorithm-name>/version --data-binary '{"settings":{"insights_enabled":true}}'

Once you’ve published your algorithm with Algorithmia Insights enabled, all executions of that specific version of the algorithm will trigger your Algorithmia Insights to be collected and reported to the Kafka broker that your Algorithmia Platform Administrator has set up.

Viewing Algorithmia Insights status for algorithm versions

The “Versions” tab of the algorithm page shows which published versions of that algorithm have Algorithmia Insights enabled:

Image of algorithm versions page with Algorithmia Insights flag highlighted