Algorithmia Insights is an algorithm metrics pipeline providing you access to your algorithm metrics payload for each algorithm session so you can measure, monitor, and manage your algorithms in production.
With Algorithmia Insights you’ll be able to opt-in to exporting your algorithm’s operational metrics to external monitoring and alerting tools. This means you’ll be able to monitor your algorithm’s execution time as well as capture any model inference metrics you expose from your algorithm’s output such as predictions or accuracy. Any metrics you opt to collect will be captured and exported in a payload and, with help from a Platform Administrator, can be connected to your external systems for monitoring and alerting.
Types of Algorithmia Insights metrics
Algorithmia Insights metrics fall into two categories:
- Operational metrics
- Inference-related metrics
Operational metrics include (but are not limited to) execution time, request ID, algorithm owner, algorithm name, algorithm version, timestamp, etc. When Algorithmia Insights are turned on for a particular algorithm version, these are automatically collected and reported without any additional action necessary on the part of the algorithm author or manager.
Inference-related metrics are defined by the algorithm author in algorithm code. Each metric is defined by a title, like
cute_cats_detected, and a numeric value, like
24. Multiple inference-related metrics can be reported at one time, and the set of metrics reported does not need to be the same from one algorithm execution to another.
One-time administrator setup
The first step in allowing Algorithmia Insights to be collected and reported for algorithms is to have the Algorithmia Platform Administrator connect your cluster to a Kafka broker / cluster to which metrics will be sent.
The Algorithmia platform supports two Kafka connection configurations:
- SCRAM (Salted Challenge Response Authentication Mechanism) over TLS. Provides encrypted communications, authentication credential database breach protection, man-in-the-middle attack protection, and internationalization support. Please see the Kafka SCRAM setup guide for configuring a Kafka broker / cluster to support this configuration.
- Unencrypted / plaintext. Provides all of the features of Algorithmia Insights for use without encryption or authentication safeguards; allows for testing / prototyping as well as use in networks with existing robust protection against attacks.
The first step to configuring your cluster for Algorithmia Insights is to navigate to the “Algorithmia Insights” administration page, which can be found under the “System Actions” section.
From here, enter the information about your Kafka broker. The following information is needed for any connection type:
- Kafka URL: a list of comma-separated Kafka bootstrap servers used to establish the initial connection to the Kafka cluster. Algorithmia will make use of all servers irrespective of which servers are specified here for bootstrapping - this list only impacts the initial hosts used to discover the full set of servers. This list should be in the form
host1:port1,host2:port2,...Since these servers are just used for the initial connection to discover the full cluster membership (which may change dynamically), this list need not contain the full set of servers (you may want more than one, though, in case a server is down).
- Topic Name: the name of the Kafka topic to which Algorithmia Insights will be published
It is the Algorithmia Platform Administrator’s responsibility to ensure that Kafka traffic (which operates over TCP) routes successfully from the Algorithmia Platform to all of the Kafka bootstrap servers in the Kafka URL list (on their specified ports) and to all of the Kafka cluster members reported by those bootstrap servers (on their specified ports).
For each execution of algorithms for which Algorithmia Insights is turned on, the Algorithmia Platform will perform multiple data send attempts to the Kafka broker / cluster over the course of four minutes. If, after that time, the data cannot be sent to the Kafka broker / cluster for any reason (Kafka server offline, failed authenticated, network disconnections, etc), the data for that specific execution will be permanently lost.
Unencrypted / plaintext
Unencrypted, plaintext connections do not require any additional configuration information.
SCRAM / TLS
Encrypted, SCRAM-authenticated connections also require:
- Username - the username used to authenticate to the Kafka cluster
- Password - the password used to authenticate to the Kafka cluster
- CA Certificate - the Certificate Authority certificate used to sign the TLS certificates that the Kafka servers use for communication.
Using Algorithmia Insights in an algorithm
Each algorithm in your Algorithmia cluster has independent settings for whether to collect and report Algorithmia Insights. Additionally, each published version of your algorithm can have Algorithmia Insights turned on or off.
Creating inference-related metrics for your algorithm
Algorithmia client libraries are used to report inference-related metrics that you create inside of your algorithm code. Please see the “Publishing Algorithmia Insights” section of the appropriate client library guide for the language that you’re using.
You can report up to 25 metrics in your code. The Algorithmia Platform will report additional metrics which do not count towards these 25.
Additionally, the total size of your metrics data (keys and values) must not exceed 25 kilobytes.
Finally, there are several reserved metric names that must not be used by client code, as they will conflict with automatically-created metrics for each algorithm execution:
Enabling Algorithmia Insights for an algorithm version
Enabling Algorithmia Insights for an algorithm version happens during algorithm publishing.
When using the web UI publishing workflow, a toggle will appear to enable Algorithmia Insights for the specific algorithm version that you are publishing.
When using the HTTP API or client libraries to publish your algorithm, passing the key value pair
"insights_enabled":true in the
settings hash will enable Algorithmia Insights for your algorithm. For example, when using cURL:
Once you’ve published your algorithm with Algorithmia Insights enabled, all executions of that specific version of the algorithm will trigger your Algorithmia Insights to be collected and reported to the Kafka broker that your Algorithmia Platform Administrator has set up.
Viewing Algorithmia Insights status for algorithm versions
The “Versions” tab of the algorithm page shows which published versions of that algorithm have Algorithmia Insights enabled: