Strata + Hadoop World 2015
Streaming Analytics: It’s Not The Same Game
We are witnessing an explosion of sensors and machine generated data. Every server, every building, and every device generates a continuous stream of information that is ever changing and potentially valuable. The existing big data paradigm requires storing data for batch analysis, and extensive modeling by a human expert, prior to deployment. This is incredibly inefficient and cannot scale. Instead there is a growing need to rapidly create adaptive models that accept streaming data sources and can take instant action.
In this talk I will describe a new paradigm for streaming data algorithms, based on recent neuroscience findings and on the computational properties of the neocortex. These systems are highly automated, adapt to changing statistics, and naturally deal with temporal data streams. They require no batch training and custom models can be deployed on the fly. Many of the core ideas have been implemented in the open source project NuPIC, and validated in commercial applications. Given the massive increase in the number of data sources, a general-purpose automated approach is the only scalable way to effectively analyze and act on continuously streaming information.