Epistemic status: This is mostly pure speculation, although grounded in many years of studying neuroscience and AI. Almost certainly, much of this picture will be wrong in the details, although hopefully roughly correct ‘in spirit’.
[Read More]
The Scale of the Brain vs Machine Learning
Epistemic status: pretty uncertain. There is a lot of fairly unreliable data in the literature and I make some pretty crude assumptions. Nevertheless, I would be surprised though if my conclusions are more than 1-2 OOMs off though.
[Read More]
Understanding Overparametrized Generalization
This is a successor post to my previous post on Grokking Grokking. Here we present a heuristic argument as to why overparametrized neural networks appear to generalize in practice, and why this requires a substantial amount of overparametrization – i.e. the ability to easily memorize (sometimes called interpolate) the training...
[Read More]
Grokking 'grokking'
Epistemic Status: This is not my speciality within ML and I present mostly speculative intuitions rather than experimentally verified facts and mathematically valid conjectures. Nevertheless, it captures my current thinking and intuitions about the phenomenon of ‘grokking’ in neural networks and of generalization in overparametrized networks more generally.
[Read More]
Clarifying Value Alignment in Predictive Processing
Recently I saw a paper by William Ratoff which argues that the alignment problem is significantly ameliorated for predictive processing (PP) agents. Their argument can be summarized as follows:
[Read More]