In this article we introduce Complementary Sparsity, a novel technique that significantly improves the performance of dual sparse networks on existing hardware. We demonstrate that we can achieve high performance running weight-sparse networks, and we can multiply those speedups by incorporating activation sparsity. Using Complementary Sparsity, we show up to 100X improvement in throughput and energy efficiency performing inference on FPGAs. We analyze scalability and resource tradeoffs for a variety of kernels typical of commercial convolutional networks such as ResNet-50 and MobileNetV2.
In this article we investigate biologically inspired architectures as solutions to catastrophic interference. Specifically, we show that the biophysical properties of dendrites and local inhibitory systems enable networks to dynamically restrict and route information in a context-specific manner. Our neural implementation marks the first time a single architecture has achieved competitive results on both multi-task and continual learning settings. Our research sheds light on how biological properties of neurons can inform deep learning systems to address dynamic scenarios that are typically impossible for traditional ANNs to solve.
In this paper, we propose that dendritic properties can help neurons learn context-specific patterns and invoke highly sparse context-specific subnetworks. Within a continual learning scenario, these task-specific subnetworks interfere minimally with each other and, as a result, the network remembers previous tasks significantly better than standard ANNs. We then show that by combining dendritic networks with Synaptic Intelligence we can achieve significant resilience to catastrophic forgetting, more than either technique can achieve on its own.
This paper reviews the state of artificial intelligence (AI) and the quest to create general AI with human-like cognitive capabilities. This review argues that improvements in current AI using mathematical or logical techniques are unlikely to lead to general AI. Instead, the AI community should incorporate neuroscience discoveries about the neocortex. It further explains the limitations of current AI techniques and focuses on the biologically constrained Thousand Brains Theory describing the neocortex’s computational principles.
This paper approaches spatial mapping as a problem of learning graphs of environment parts. We show that hippocampal modules may dynamically create graphs representing spatial arrangements, and this proposed fast-relation-graph-learning algorithm can expand to incorporate many spatial and non-spatial tasks.
