The Problem of Representation
Years ago, a retiring artificial intelligence researcher told Jeff Hawkins that "one of biggest problems in AI–no, the only problem in AI–is the problem of representation." If the meaning of that statement isn’t intuitively obvious to you, don’t worry. It wasn’t immediately clear to Jeff either.
Jeff explained how he came to understand and address this problem in a recent keynote address at the International Symposium on Computer Architecture. We posted a video of the keynote on YouTube. In this talk, Jeff describes the brain as a predictive modeling system that takes streams of input from the senses and learns sequences in real time. The brain represents its inputs and the state of its world via the activity of neurons. At any point in time most of the neurons are inactive and a few active, thus the brain’s representations are “sparse.” Consequently we call these "Sparse Distributed Representations” (SDRs) in our algorithms. SDRs exhibit unique properties that enable benefits such as semantic generalization and robustness to errors. These properties, in turn, allow the brain to learn about objects and how they relate to each other holistically, without the programmatically defined data structures used in computers. By applying these principles in products like Grok, we can finally address the problem of representation.
The talk ends with Jeff’s speculation on how the technology will evolve. Grok itself is merely the first iteration of this technology. Unlocking the operating principles of the neocortex will not necessarily culminate in solutions addressing the “classic” AI problems of vision, language and speech. The history of technology suggests that truly revolutionary technological advancements develop in ways that even its inventors could never imagine. I’m reminded of Alexander Graham Bell’s prediction, “One day there will be a telephone in every major city in the USA.”
If you’d like to know more, make sure to watch the video.