Intellectual Progress in 2020

The yearly intellectual progress report is a little late this year due to having to write up a PhD. Now that the PhD thesis beast is at least somewhat tamed, in a breather, I can try to figure out what progress I have made this year.

As expected, ‘extracurricular’ intellectual progress is somewhat down this year compared to last, since I’ve been focusing on finishing up my PhD and getting out workable papers. This year I have primarily gone for depth over breadth and have put out many papers.

In terms of academic output I have been reasonably productive – with about 6 full or equal authorship papers out – about 1 every two months – and several more that are just in the final stages of getting sent out. This level of productivity has rescued my PhD and put me in a reasonable position. So that is good and it has been a fun ride actively contributing to multiple fields. I have contributed to active inference and (model-based) deep reinforcement learning and gained a deep and solid understanding there both in terms of the implementation and the theory. I feel like my understanding of reinforcement learning is substantially deeper than it was last year, primarily due to more practical, hands-on experience. Secondly, I have thought a lot about the mathematics of various variational objectives for control and understand frameworks like control as inference at a very deep level so they are no longer intimidating to me. I have learnt to become a master of arbitrary and strange KL divergence manipulations, which is a niche skill, but has revealed some interesting facts.

I have also studied deeply the nature of credit assignment in the brain. This is a fascinating area and could have real implications for AI timelines and our understanding of how intelligence and learning is implemented in the brain. I fully understand the rate-coded neuron model literature, and have contributed several new algorithms here. I feel like I have mastered and exhausted much of the potential of these ratecoded models, and my goal now is to focus on mroe neurobiologically realistic spiking networks and also attack the main problem, which I’m becoming increasingly convinced is that of credit assignment through time, not space, as space is quite straightforwardly solved in many cases. This is an area where I hope to contribute more in the coming year.

Also, this year I have finally got to the bottom of the free energy principle and now understand it all at quite some level of detail – for instance, thanks to the reading group, I now feel like I understand the core arguments of particular physics which is a huge advance for me personally. In that sense, I feel like I’ve finally got to the forefront of the free energy principle and active inference field. Now the next steps are to leapfrog beyond it and pursue my own path in the future. I also feel like I finally understand all the process theories under the FEP (predictive coding, discrete state space active inference, DEM and generalized coordinates etc). I would say that I am pretty much at a world expert level in the FEP and related theories, and rapidly approaching that in some other areas – specifically deep model based reinforcement learning as well as algorithms for biologically plausible credit assignment. Thus, in a weird sense, I have actually achieved the goal of a PhD, which is to get up to the state of the art and make a contribution to at least one field, which I finally believe I have done.

Finally, I have figured out pretty precisely the origin of information gain terms in exploration objectives. This means I can understand where intrinsic terms in artificial curiosity arise from and why you can’t get them simply from a Bayesian control objective such as control as inference. This was a major source of confusion for me and my research group throughout this year and I have eventually figured it out pretty precisely, including understanding a lot about the expected free energy, generalized free energy, and other topics which has been super fascinating to figure out, although once I figured it out it all seems a bit trivial now. It’s funny how that goes. Paper on this will (hopefully) be out soon.

Those are the key research results I have come across this year. In terms of general skills, I feel I have advanced significantly in understanding how to do effective research and have developed a much better ‘nose’ for figuring out what is a promising research direction or not. This has been substantially due to the fantastic and constant feedback of the research group at Sussex, whereas the year before last I was largely isolated. I think this gets to the overriding importance of mentorship and the fact that effective action is much more important than simply hard work in the vast majority of cases. Since most hard work, if not effectively directed, is largely wasted. It’s also beeen very interesting coming to the end of my PhD and looking back on the journey it has been over the last three years. I’ll have another longer post discussing this when the thesis is actually (finally!) submitted.

Extracurricular studies have been relatively neglected this year – especially my mathematical maturity which I wanted to strongly develop and largely have not, besides a course on real analysis and a course on differential geometry in general relativity I only got part way through. Nevertheless I have struck out into the fascinating fields of systems biology, and have taken a few biology courses which have begun to open my mind to the potentials in this field, although I am definitely a long way away from the intellectual mastery of this field which I wish to attain ultimately. Secondly, I have also seriously studied economics for the first time, including courses on microeconomics, macroeconomics, finance and money and banking which, especially the finance and the money and banking courses have been substantially eye-opening. Whiel further economics study is likely to lead to lower marginal gains than these core subjects, I suspect that doing some study of this next year, and branching out into more practical business courses to get that side of the story will be extremely helpful as well. Nevertheless, foundational lecture series have been much less common this year, potentially also due to running out of core subjects in machine learning and AI to study. Nevertheless, over the course of the year, primarily due to my paper studies, I have gained a fairly strong understanding of additional core ML architectures such as attention and transformers, neural turing machines, and other memory based approaches as well, which may prove useful.Overall though, my actual machine learning implementationa experiences remain relatively poor and I have not yet attained a SOTA result. I have focused primarily on the mathematical underpinnings of deep RL, on credit assignment and predictive coding, and on exploration in model-based reinforcement learning methods, but not seeking to match the state of the art. This is primarily due to computational and personal skill limitations. A secondary and important goal over the next year is to try to solve these current limitations and actually attain real state of the art ML capabilities instead of just nibbling around the edges. Also, I have discovered einsum, which is amazing.

In terms of blog posts, I have done substantially better than last year, although not as well as in 2017 when I really started blogging, and have ported most of those blog posts across. In the next year 2021 I need to get substantially better with writing serious blog posts and articles which can both encompass pieces of research too small to publish, as well as other takes and interesting phenomena.

Below is a list of extracurricular learning for 2020.

January – None

Feb – Real Analysis and Topology

March

– Systems Biology – Systems Medicine

April – None

May – Differential Geometry and General relativity (only part way through).

June None

July Microeconomics Macroeconomics

August Finance

September Money and Banking Evolution and Ecology

October – None