Sparse distributed representations
One of the most remarkable observations about the neocortex is that no matter where you look, the activity of neurons is sparse, meaning only a small percentage of them are spiking at any point in time. The sparsity might vary from less than one percent to several percent, but it is always sparse. Sparse distributed representations (SDRs) model this property and are a key component of HTM theory .
Our research on SDRs is aimed at uncovering properties of sparse representations that provide insight into the neocortex. The research also provides a practical mathematical framework that helps explain some of the numbers and parameters used in HTM systems.
Our research has focused on binary sparse distributed representations, i.e. an array consisting of a large number of bits where a small percentage are 1’s and the rest are 0’s. The bits of SDRs correspond to neurons in the brain, where 1 represents an active neuron and 0 an inactive one. Each bit typically has some meaning (such as the presence of an edge at a particular location and orientation). The information is distributed amongst the bits such that no single bit is critical.
How accurately can neurons recognize sparse patterns?
Through active dendrites, a neural property that is a key underpinning of HTM theory, neurons are thought to detect patterns using a small number of synapses (as few as 10 to 20 synapses). Given that neural populations are large, and their activity is extremely unreliable and noisy, how could neurons reliably detect patterns with such a tiny number of synapses? The above scaling laws show that under a range of numbers, even 10 to 20 synapses can accurately classify patterns under very noisy conditions. The equations can even be used to explain the experimentally observed thresholds in active dendrites (see Cosyne poster below).
Sparse Distributed Representations Resources
 Cui, Y., Ahmad, S., & Hawkins, J. (2017) The HTM Spatial Pooler – a neocortical algorithm for online
sparse distributed coding. Front. Comput. Neurosci., 11, 111.  Ahmad, S. & Hawkins, J. (2016) How do neurons operate on sparse distributed representations?
A mathematical theory of sparsity, neurons and active dendrites. arXiv, arXiv:1601.00720.  Hawkins, J. et al. (2016) Biological and Machine Intelligence. Sparse Distributed Representations chapter.
- Sparse Distributed Representations - Our Brain’s Data Structure
- SDR Capacity & Comparison (HTM School, Episode 2)
- SDR Overlap Sets and Subsampling (HTM School, Episode 3)
- SDR Sets & Unions (HTM School, Episode 4)
- Spatial Pooling: Input Space & Connections (HTM School, Episode 7)
- Spatial Pooling: Learning (HTM School Episode 8)
- Boosting (HTM School, Episode 9)