Our VP of Research and Engineering Subutai Ahmad is speaking at the SigOpt AI & HPC Summit on November 16th, 2021. He will present novel sparse networks that achieve high accuracy and leverage sparsity to run 100x faster than their dense counterparts.
The SigOpt Summit brings together modelers across a wide variety of industries and modeling problems to discuss their approach to experimentation and how this translated to better modeling results. Attendees and speakers include data scientists, AI leaders, AI platform engineers, researchers, machine learning engineers and deep learning engineers. Register for free here.
Abstract:
Most deep learning networks today rely on dense representations. This stands in stark contrast to our brains which are extremely sparse, both in connectivity and in activations. Implemented correctly, the potential performance benefits of sparsity in weights and activations is massive. Unfortunately, the benefits observed to date have been extremely limited. It is challenging to optimize training to achieve highly sparse and accurate networks. Hyperparameters and best practices that work for dense networks do not apply to sparse networks. In addition, it is difficult to implement sparse networks on hardware platforms designed for dense computations. In this talk we present novel sparse networks that achieve high accuracy and leverage sparsity to run 100X faster than their dense counterparts. We discuss the hyperparameter optimization strategies used to achieve high accuracy, and describe the hardware techniques developed to achieve this speedup. Our results show that a careful evaluation of the training process combined with an optimized architecture can dramatically scale deep learning networks in the future.
Subutai Ahmad • CEO