In the age of Big Data, algorithms give companies a competitive advantage. Today’s most important technology companies all have algorithmic intelligence built into the core of their product: Google Search, Facebook News Feed, Amazon’s and Netflix’s recommendation engines.
“Data is inherently dumb,” Peter Sondergaard, senior vice president at Gartner and global head of Research, said in The Internet of Things Will Give Rise To The Algorithm Economy. “It doesn’t actually do anything unless you know how to use it.”
Google, Facebook, Amazon, Netflix and others have built both the systems needed to acquire a mountain of data (i.e. search history, engagement metrics, purchase history, etc), as well as the algorithms responsible for extracting actionable insights from that data. As a result, these companies are using algorithms to create value, and impact millions of people a day.
“Algorithms are where the real value lies,” Sondergaard said. “Algorithms define action.”
For many technology companies, they’ve done a good job of capturing data, but they’ve come up short on doing anything valuable with that data. Thankfully, there are two fundamental shifts happening in technology right now that are leading to the democratization of algorithmic intelligence, and changing the way we build and deploy smart apps today:
- The Algorithm Economy
The confluence of the algorithm economy and containers creates a new value chain, where algorithms as a service can be discovered and made accessible to all developers through a simple REST API. Algorithms as containerized microservices ensure both interoperability and portability, allowing for code to be written in any programming language, and then seamlessly united across a single API.
By containerizing algorithms, we ensure that code is always “on,” and always available, as well as being able to auto-scale to meet the needs of the application, without ever having to configure, manage, or maintain servers and infrastructure. Containerized algorithms shorten the time for any development team to go from concept, to prototype, to production-ready app.
Algorithms running in containers as microservices is a strategy for companies looking to discover actionable insights in their data. This structure makes software development more agile and efficient. It reduces the infrastructure needed, and abstracts an application’s various functions into microservices to make the entire system more resilient.
The Algorithm Economy
The “algorithm economy” is a term established by Gartner to describe the next wave of innovation, where developers can produce, distribute, and commercialize their code. The algorithm economy is not about buying and selling complete apps, but rather functional, easy to integrate algorithms that enable developers to build smarter apps, quicker and cheaper than before.
Algorithms are the building blocks of any application. They provide the business logic needed to turn inputs into useful outputs. Similar to Lego blocks, algorithms can be stacked together in new and novel ways to manipulate data, extract key insights, and solve problems efficiently. The upshot is that these same algorithms are flexible, and easily reused and reconfigured to provide value in a variety of circumstances.
For example, we created a microservice at Algorithmia called Analyze Tweets, which searches Twitter for a keyword, determining the sentiment and LDA topics for each tweet that matches the search term. This microservice stacks our Retrieve Tweets With Keywords algorithm with our Social Sentiment Analysis and LDA algorithms to create a simple, plug-and-play utility.
The three underlying algorithms could just as easily be restacked to create a new use case. For instance, you could create an Analyze Hacker News microservice that uses the Scrape Hacker News and URL2Text algorithms to extract the text for the top HN posts. Then, you’d simply pass the text for each post to the Social Sentiment Analysis, and LDA algorithms to determine the sentiment and topics of all the top posts on HN.
The algorithm economy also allows for the commercialization of world class research that historically would have been published, but largely under-utilized. In the algorithm economy, this research is turned into functional, running code, and made available for others to use. The ability to produce, distribute, and discover algorithms fosters a community around algorithm development, where creators can interact with the app developers putting their research to work.
Algorithm marketplaces function as the global meeting place for researchers, engineers, and organizations to come together to make tomorrow’s apps today.
Containers are changing how developers build and deploy distributed applications. In particular, containers are a form of lightweight virtualization that can hold all the application logic, and run as an isolated process with all the dependencies, libraries, and configuration files bundled into a single package that runs in the cloud.
“Instead of making an application or a service the endpoint of a build, you’re building containers that wrap applications, services, and all their dependencies,” Simon Bisson at InfoWorld said in How Containers Change Everything. “Any time you make a change, you build a new container; and you test and deploy that container as a whole, not as an individual element.”
Containers create a reliable environment where software can run when moved from one environment to another, allowing developers to write code once, and run it in any environment with predictable results — all without having to provision servers or manage infrastructure.
This is a shot across the bow for large, monolithic code bases. “[Monoliths are] being replaced by microservices architectures, which decompose large applications – with all the functionality built-in – into smaller, purpose-driven services that communicate with each other through common REST APIs,” Lucas Carlson from InfoWorld said in 4 Ways Docker Fundamentally Changes Application Development.
The hallmark of microservice architectures is that the various functions of an app are unbundled into a series of decentralized modules, each organized around a specific business capability.
Martin Fowler, the co-author of the Agile Manifesto, describes microservices as “an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.”
By decoupling services from a monolith, each microservice becomes independently deployable, and acts as a smart endpoint of the API. “There is a bare minimum of centralized management of these services,” Fowler said in Microservices: A Definition of this New Architectural Term, “which may be written in different programming languages and use different data storage technologies.”
Similar to the algorithm economy, containers are like Legos for cloud-based application development. “This changes cloud development practices,” Carlson said, “by putting larger-scale architectures like those used at Facebook and Twitter within the reach of smaller development teams.”
- The algorithm economy and containers are changing the way developers build and ship code.
- The algorithm economy allows for the building blocks of algorithmic intelligence to be made accessible, and discoverable through marketplaces and communities.
- Containerizing algorithms enables them to be packaged as microservices, making them accessible via an API, and hosted on scalable, serverless infrastructure in the cloud.
It’s been an exciting and incredibly productive year here at Algorithmia. As we kickoff 2016, I want to look back at everything we accomplished in 2015.
First, I want to thank our community of more than 15,000 algorithm and application developers for their support – it has been an amazing experience. We’re honored by everybody that signed-up, used us in your applications, and added algorithms to the platform. We’re truly thankful for this unique community.
These are the major milestones we crossed in 2015:
Algorithmia leaves private beta and publicly launches, introducing a community built around the algorithm economy, where state-of-the-art algorithms are always live, and accessible by everyone. The algorithm marketplace launches with over 800 algorithms, 3,500 users, and an API for leveraging the building blocks of human intelligence in your apps.
How to Build Your Own Google
Algorithms on Algorithmia are like Legos, where they can be mixed and matched to explore the web algorithmically. In this demo, we use Algorithmia to implement PageRank, the algorithm Google was originally based on, to crawl a site, retrieve it as a graph, analyze the connectivity of the graph, and then analyze each node for its contents.
Using Artificial Intelligence to Detect Nudity
We combine artificial intelligence with University research to teach an algorithm to determine if there is nudity in an image. The result is our Nudity Detection algorithm, which uses a combination of nose, face, and skin color detection algorithms to identify if there are people in an image, and if any of them are nude.
Don’t believe us? Try out the demo at isitnude.com.
Navigate Product Hunt Like a Pro
We build a Chrome Extension that uses FP-Growth and Keyword Set Similarity algorithms to surface related products on Product Hunt from users who’ve upvoted the product you’re browsing (i.e. collaborative filtering). Get the Chrome Extension here, and start discovering better products on Product Hunt today.
AWS Lambda Partnership
Algorithmia teams up with AWS Lambda, enabling developers to build intelligent, serverless apps in minutes by leveraging our built-in Node.js blueprint. In this demo, we show you how to quickly make a serverless photo app to create digital art in less than 300 lines of code using AWS Lambda and Algorithmia.
The #1 Big Data Startup
Supporting the Innovators of Tomorrow
Algorithmia joins more than 600 student hackers at DubHacks, the largest collegiate hackathon in the Pacific Northwest, as sponsor and participant to help teams build projects, and create solutions to real-world problems.
Content Recommendations Made Simple
We launch Algorithmia Recommends, a free content recommendation plugin for any WordPress or Drupal blog, or website to increase their engagement and retention. Algorithmia Recommends is built on the Breadth First Site Map web crawler algorithm, and powerful natural language processing algorithms Keywords For Document Set and Keyword Set Similarity to find and categorize all the pages on your website in order to help your users find content that’s most relevant to their interests.
Free eBook Published
Algorithmia launches its first eBook, Five Algorithms Every Web Developer Can Use and Understand, which teaches you how to harness the power of algorithms so you can make every app a smart app. In this short primer, we cover PageRank, Language Detection, Nudity Detection, Sentiment Analysis, and TF-IDF, and how you could implement it today.
GeekWire: Algorithmia Top Seattle Startup
Algorithmia joins GeekWire’s prestigious, annual list of the 10 most promising early-stage startups in the Seattle area. Our business model gets translated onto a giant six-foot by six-foot cocktail napkin, which is unveiled during GeekWire Gala at the Museum of History & Industry (MOHAI).
One of the 10 Coolest Big Data Startups
Algorithmia caps off an incredible year with another award, this time CRN names us one of the coolest big data startups.
This could have not been achieved without our amazing team Anthony, Besir, John, Jonathan, Kenny, Liz, Matt, Patrick and Zeynep. As well as our awesome 2015 interns Ahmad and Mark.
We’re just getting started, and 2016 is shaping up to be even bigger as we add more algorithms and use cases.
We’re looking forward to another incredible year, and we’re excited to have you along for the journey.
– Diego M. Oppenheimer, CEO
A year ago, we shared with you our belief that Algorithm Development is Broken, where we laid out the problems in the world of algorithm development, and our vision of how to make things better.
Since then, we’ve been working hard to create our vision of the world’s first open marketplace for algorithms – and you, the developers of the world, have enthusiastically joined us to help active the algorithm economy.
Today, we are proud to open our doors to the world, and deliver on our promise of a global, always-available, open marketplace for algorithms. For the first time, algorithm developers can publish their algorithms as a live API and applications can be made smarter by taking advantage of these intelligent algorithms.
Algorithm Development with Algorithmia
- Live: easy access to any algorithm, callable at any time
- No dependencies: no downloads, no installs, no stress
- Interoperable: call any algorithm regardless of programming language
- Composable: algorithms are building blocks, build something bigger!
- Invisible but powerful infrastructure: don’t worry about containers, networks, or scaling; we make your algorithm scale to hundreds of machines
- No operations: we scale, meter, optimize and bill for you
With our public launch we are enabling payments and go from being a repository of algorithms, to a true marketplace:
- For algorithm developers, Algorithmia provides a unique opportunity to increase the impact of their work and collect the rewards associated with it.
- For application developers, Algorithmia provides the largest repository of live algorithms ever assembled, supported by a vibrant community of developers.
So far, users have contributed over 800 algorithms, all under one API that grows every day. The most exciting thing from our point of view is watching the Algorithmia Marketplace grow and expand in its functionality.
In just the last few months, Algorithmia has gained the ability to:
- Summarize and generate topics from unstructured text
- Extract structure and information from arbitrary websites
- Perform computer vision tasks like Face Detection and Image Similarity
- Look for patterns and anomalies in data
- Dozens of microservice building blocks
- … with more added every day
Thanks to the thousands of developers who joined our cause, contributed algorithms, and gave us countless hours of feedback so far. We are very excited to be opening our doors, and we look forward to helping you create something amazing!
Interested? Join the largest marketplace for algorithms in the world.
Highly complex algorithm development is often at the core of innovative software products. The most famous algorithm of the last decade led to the development of the $396 billion dollar titan Google, which changed our lives with a simple search box.
Today, an estimated 80 percent of equity trading in the US is done using automated algorithms. Even our phones use algorithms to figure out where we are, what we are doing, and what we might want to do next. A few days ago, DeepMind, a company that developed the next generation of artificial intelligence algorithms, was bought for a record sum. These examples are just scratching the surface of what algorithms can do.
As the volumes of data we collect on a day-to-day basis grow, data architects are on a race to have more, better, and faster hardware; yet Moore’s Law is showing a slow down, indicating that throwing more hardware at a data problem is no longer going to fulfill the needs of organizations. As companies are increasingly and continuously hungry for understanding of how they operate and how they can improve, they must concentrate more on the efficiency and quality of the underlying algorithms that compose suites of data analysis packages versus the hardware they run on.
The Current State of Algorithm Development
It is safe to say that on any given day there are thousands computer scientists developing the state of the art and pushing the limits of software, yet therein lies a serious problem: Algorithms are being developed all the time but are not getting into the hands of people and applications that could benefit from them.
- A vast majority of algorithmic developments occurs in one of two places: either private research and development programs (e.g., Microsoft Research, Google, etc.) or in one of more than 320+ academic institutions with computer science departments around the world. The output of these types of research and development is almost always private white papers or academic papers.
- Companies and application developers have a hard time taking advantage of advanced algorithms. Academic papers are hard to find and even harder to implement. Even in the best case, when published code is available, more time is spent setting up infrastructure and fighting with undocumented libraries and dependencies than actually doing any algorithm development.
For example, a while back I was researching Latent Dirichlet Allocation. In layman’s terms, it is an algorithm that allows for the extraction of topics from documents without an understanding of the language itself. A quick search on Google brought up a dozen research papers and a couple of libraries where this algorithm had been implemented. So where do I start? I guess I could go ahead and implement one of the libraries, but which one? Do I even know if this is going to work? Once I do figure it out will it work in my current system?
The Birth of Algorithmia
Algorithmia was born out of frustration with these types of problems and the current state of algorithm development and deployment. Algorithmia is moving away from developers working in isolation and toward providing a community for algorithm developers to share knowledge, test algorithms, and run them directly in their applications. How we’re different is that every algorithm is live, so there will be no more time spent finding the right libraries, compilers, and/or virtual machines—Algorithmia takes care of all of that for you. Find a useful algorithm in our API? Great, use that in your own applications.
Algorithmia is a live, crowd-sourced algorithm API. Our goal with Algorithmia is to make applications smarter, by building a community around algorithm development, where state-of-the-art algorithms are always live and accessible to anyone.
Intrigued? Come check us out.