Amazon Pushes the Boundaries of Cloud Data Science at re:Invent 2015

As an artificial intelligence and machine learning expert, I‘ve watched with keen interest as cloud computing platforms like Amazon Web Services (AWS) have evolved to support the growing demands of data science workloads. Sophisticated modeling techniques, ever-increasing volumes of data, and the need for scalable, on-demand computing resources are driving more and more data science into the cloud.

So I was excited to attend AWS re:Invent 2015 and see Amazon‘s latest and greatest offerings for data scientists and big data professionals. The conference did not disappoint, with AWS unveiling a range of new services and enhancements squarely aimed at making data science in the cloud faster, easier, and more powerful than ever before.

Diving into the Data Science Announcements

The headliner among the data science-related announcements was undoubtedly AWS IoT, a new managed cloud platform for building and deploying applications for the Internet of Things (IoT). AWS IoT aims to make it easy to connect and manage billions of heterogeneous IoT devices, ingest the massive volumes of data they generate, and build real-time IoT applications.

At its core, AWS IoT consists of a device gateway for securely registering and communicating with connected devices, a rules engine for processing and routing device data, a registry for indexing devices and metadata, and SDKs for building device software. It offers tight integration with AWS‘s existing big data and analytics services, allowing IoT data to flow seamlessly into tools like Amazon Kinesis, AWS Lambda, Amazon S3, Amazon Machine Learning, and Amazon DynamoDB for processing and analysis.

High-level architecture of the AWS IoT platform (Source: AWS)

From a data science perspective, AWS IoT provides the foundational components needed to collect, store, and run analytics on IoT data at massive scale in the cloud. By encapsulating the underlying infrastructure as managed services, it enables data scientists to focus on mining IoT data for actionable insights rather than provisioning servers and managing data pipelines.

Real-world use cases could include predicting equipment failures from sensor data, optimizing vehicle routes and fuel consumption, or identifying anomalous behavior in smart meters. Early customers like Sonos, Scout, and Pentair are already using AWS IoT to power data-driven IoT applications and services.

Another headline announcement was Amazon QuickSight, a new business intelligence (BI) service that harnesses the power of machine learning to enable blazingly fast, easy-to-use data visualization and ad hoc analysis. QuickSight aims to democratize BI and put powerful analytics capabilities into the hands of every employee, at a radically lower cost than traditional BI solutions.

An example Amazon QuickSight dashboard (Source: AWS)

The secret weapon powering QuickSight is a new query engine called SPICE (Super-fast, Parallel, In-memory Calculation Engine). SPICE uses a combination of columnar storage, in-memory technologies, and machine code generation to accelerate analytical queries and enable near-real-time interaction with large datasets.

When users upload data into SPICE, QuickSight‘s AutoGraph feature uses machine learning to automatically infer the optimal chart type and visualization for the data based on its characteristics. Users can ask questions of their data in natural language, and QuickSight will translate those questions into queries and generate relevant visualizations on the fly.

For data scientists, QuickSight offers a powerful yet easy way to explore data, validate hypotheses, and share insights with others. And at only $9 per user per month, it‘s an order of magnitude cheaper than solutions like Tableau and QlikView, making large-scale BI rollouts economically feasible even for smaller organizations.

Behind the scenes, QuickSight integrates with Amazon‘s ecosystem of data management and processing services, allowing it to directly query data in Amazon Redshift, Amazon RDS, Amazon Aurora, Amazon S3, and more without time-consuming ETL. Third-party data sources like Salesforce are also supported.

The next major announcement came in the realm of real-time streaming data with Amazon Kinesis Firehose. Firehose is a fully managed service for ingesting streaming data and reliably loading it into AWS data stores like Amazon S3, Amazon Redshift, or Amazon Elasticsearch Service.

With Firehose, users simply create a "delivery stream", configure where they want the data to be delivered, and then start pushing data to the stream from mobile apps, web apps, IoT sensors, or any other source. Firehose automatically takes care of all the scaling, buffering, and error handling required to continuously process the data.

Data flow in Amazon Kinesis Firehose (Source: AWS)

This greatly simplifies the process of capturing and ingesting streaming data, which has traditionally required complex infrastructure setup and ongoing operational overhead. Instead, data scientists can focus their efforts on running analytics on the data and extracting timely insights.

Some interesting use cases that Firehose enables:

• Capturing and analyzing clickstream data in real-time to optimize ad placements or personalize content
• Ingesting application logs for troubleshooting, security monitoring, and auditing
• Persisting IoT sensor data into a data lake for long-term storage and batch processing

To illustrate the potential, Hearst Corporation is using Kinesis Firehose to ingest data from hundreds of newspaper and magazine websites into Amazon S3 and Amazon Redshift, where it can then be analyzed to provide real-time content recommendations to readers.

The AI/ML Perspective

As someone who spends their days building machine learning models and intelligent applications, I‘m always on the lookout for tools and services that can increase the velocity and agility of AI/ML projects. The data science announcements from re:Invent 2015 are compelling because they allow data scientists to offload undifferentiated heavy lifting to AWS‘s battle-tested, highly scalable managed services.

Take AWS IoT for example. Building a secure, reliable device gateway and real-time data ingestion pipeline is a complex engineering undertaking. By providing these as managed services through AWS IoT, Amazon is enabling data science teams to shortcut the infrastructure phase and jump straight to training models and building intelligent applications on top of IoT data. This is a huge unlock in terms of time-to-value.

Amazon QuickSight is another great example. One of the biggest challenges in democratizing ML and data science is making the outputs easily consumable for non-technical stakeholders. QuickSight‘s natural language query capabilities and automatic visualization features go a long way towards making data insights more accessible to business users. And its cost-effective, cloud-native architecture means those insights can be delivered at enterprise scale.

Looking more broadly, the AWS data science announcements point to an important paradigm shift that‘s happening in the world of enterprise AI/ML. The conventional approach has been to invest in on-premises infrastructure and download open source software to piece together a bespoke data science environment. But with the rapid pace of innovation in cloud AI/ML services, there are now significant advantages to going "all-in" on a cloud platform like AWS.

Managed services for data ingestion, storage, processing, and visualization eliminate a lot of the foundational work required to prepare data for ML. Elastic compute resources allow data scientists to rapidly scale up and down ML training and inference jobs based on demand. And the broad integration between AWS services makes it possible to build end-to-end, closed-loop intelligent applications without complex custom integration.

Building on AWS also ensures data science teams are always working with the latest and greatest in terms of tools and technology. AWS is continuously rolling out new features and services to increase the efficiency and sophistication of cloud-based ML workflows. Two great examples are Amazon Machine Learning, which uses automated model tuning and selection to allow developers to add ML capabilities to their apps without learning complex ML theory, and AWS Lambda, which provides a serverless environment for running ML inference at scale.

Conclusion

After spending a few days at AWS re:Invent, one thing was abundantly clear: Amazon is deeply invested in pushing the boundaries of what‘s possible with data science and AI/ML in the cloud. And they‘re putting their money where their mouth is, if the slew of new data science offerings announced is any indication.

What‘s most exciting to me is how these new offerings will democratize access to cutting edge tools and accelerate the delivery of intelligent applications and services. Data science is no longer confined to the ivory towers of Ph.D researchers and well-funded tech companies – it‘s rapidly becoming something that developers and decision-makers of all technical abilities and resource levels can tap into.

AWS re:Invent 2015 provided a compelling glimpse into a future of ubiquitous AI/ML, where data-driven intelligence is woven into the fabric of every application, business process, and customer interaction. A future where data science at enterprise scale is achievable for every organization, thanks to the power and flexibility of the cloud.

Amazon has fired a major salvo in the race to become the preeminent cloud platform for data science, and I for one can‘t wait to see what innovations emerge as a result. One thing is for sure, it‘s never been a more exciting time to be working at the intersection of data science and cloud computing. The sky‘s the limit.