On 9 June 2020, Algorithmia CTO, Kenny Daniel, co-hosted a webinar with Sam Charrington of TWIML on building internally versus buying an existing machine learning operations platform. The webinar recording can be accessed here. The discussion generated a lot of questions, and Kenny, Sam, and other Algorithmia leaders compiled them and provided answers in a follow-up email. The Q&A is helpful for anyone considering machine learning development tactics at their organization. 

The build versus buy webinar questions


Does your platform integrate well with auto-ml services?

Algorithmia: Yes, we work with many auto-ml services, including DataRobot models, H20 Driverless AI and others. We have rich integration APIs that make it simple to use the training platform of your choice with Algorithmia. 


Does the low importance on reducing bias indicate that perhaps people are using ML with the wrong intention?

This can allow banks to refuse approving mortgages more quickly, but doesn’t address whether they’re actually approving mortgages to the right people and pricing the rates correctly.

Sam Charrington: Agreed, there could be a lot of reasons for this. I’m assuming it’s more related to a “hierarchy of needs” and organizations will eventually get to this. That said, I’ve found the traditionally regulated industries, including financial services, to be much farther ahead in this regard.


What are approaches to measure the performance (with and without retraining regularly) when looking at changing environments/systems/ processes?

Say we have already deployed models and we want to ensure continuous quality by continuously validating the model—not only on historical data, but also on live data. If we have a productively running system, we can’t always assess whether a prediction or decision of a model was wrong (for instance if a model decides to cancel a billing process we never know if the billing process would have been successful or not).

Algorithmia: The types of monitoring and testing will depend on the nature of the ML problem, the data available, and what labeled examples you have. You need a way to know if the predictions were right or wrong. Sometimes this happens later, either by users themselves flagging classifications as right or wrong. Or it could be through a labeling service. You can also in some cases compare to a previous version of a model, if the previous model disagrees, it might be worth having a human label it. Finally, some algorithms output a confidence score in its prediction, if that confidence level drops, it could be an indication that the model or the data are mismatched.  For more information on measuring ML operations, check out this video of our webinar on 10 Measures and KPIs for ML success.


Would you mind sharing statistics on the respondents of the TWIML survey?

Sam Charrington: We’ll be sharing the results when the ML Pulse 2020: ML Development, Deployment and Operations Survey is ready for distribution. Sign up to receive the report here.  


Do you support batch processing?

Algorithmia: Yes, we provide a flexible system in which you can perform individual, serial-batch, or parallel-batch prediction using the same model. More technical details on how we support batch processing can be found on our Developer Center here.


If I buy an off-the-shelf ML management platform, do I need to hire full-time ML deployment and maintenance/technical support staff? 

Algorithmia: For the management of Algorithmia, your team should include a fractional DevOps resource who is familiar with your IT policies, infrastructure knowledge of your cloud platform, and ability to engage once a quarter for upgrades. This averages at 5 hours every quarter for most of our enterprise customers. Algorithmia was designed with existing IT resources in mind, to support the specialized needs of the ML lifecycle, but do so with the constraints of existing software and data lifecycle that IT is familiar with. 


Do you think the high resource requirements of Deep Neural Networks is a considerable concern or even hindering businesses to deploy models?  

Algorithmia: The heavy compute requirements are one of the defining factors in making ML more difficult to deploy than traditional software. That being said, with resources available elastically in the cloud, the heavy compute mostly just translates to managing cost for performance. Managing the consumption of compute resources and their associated costs is one of the challenges we built Algorithmia to address.


Would fairness be included as a governance metric? So we don’t introduce unfair models that continue to be built upon and become the de facto standard?

Algorithmia: Organizations should consider leveraging existing oversight processes (e.g. internal audit) and committees, including independent observers to monitor implementations of AI/ML capabilities and applications. Given AI/ML techniques will be used across multiple data sets and applications, providing aggregated reporting across the enterprise, from both development and operations will help organizations communicate with internal and external stakeholders on aspects like fairness and conduct.

While fairness itself is not a metric, data from operational platforms like Algorithmia when combined with developmental data can help answer and explain how models are being used and applied.


Can models be rationalized down to fewer high impact ones enabling better targeting of multiple business issues?

Algorithmia: Is it better to have fewer consolidated models? Or more compartmentalized models that can be pipelined together? Organizations will have to answer these questions for themselves depending on their application. Platforms like Algorithmia support both. In most cases, organizations that choose to leverage model pipelining will be able to reuse smaller models more efficiently to produce complex results faster decouple iteration for more focused improvement of the models.


Could you discuss how an organization should navigate the decision of which platform to go with (ie. Algorithmia v AWS SageMaker)?

Algorithmia: ML operations and management platforms, like Algorithmia, focus on supporting the deployment and operations phases of the ML lifecycle. Using APIs to integrate with a host of ML and DS developmental tools, languages and frameworks, as well as integrating with an organization’s existing security and governance applications. For example you can train a model in SageMaker, as well as many other platforms, run them on Algorithmia and the infrastructure of your choice.

Many other tools are tied to the development process and limit the options available to Data scientists, effectively locking them into a limited technology stack. More information on the Challenges organizations face with tackling ML operations and management can be found in our ebook here.