Epistemic status Still trying to work out my thoughts on this. Things change pretty regularly. My current thinking on technical AI safety questions and threat models likely diverges by now reasonably far from the LW median.
[Read More]
Linear Attention as Iterated Hopfield Networks
In this short note, we present an equivalence and interpretation of the recurrent form of linear attention as implementing a continually updated hopfield network. Specifically, as the recurrent transformer is performing generation for each token, it simply adds a continuous ‘memory’ via Hebbian plasticity to a classical continuous hopfield network...
[Read More]
Learning Linear Representations through Implicit Subspace Selection
Epistemic status: Highly speculative, basically shower thoughts. These are some thoughts I had a few months back but just got motivation to write them up today.
[Read More]
Intellectual Progress in 2023
2023 has also been an interesting year. The first half of the year was at Conjecture with a brief stint cofounding Apollo and then cofounding a soon-to-be-revealed (with any luck) startup which I shall have to remain fairly quiet on for now. There has been lots of change and personal...
[Read More]
Open source AI has been vital for alignment
Epistemic Status: My opinion has been slowly shifting towards this view over the course of the year. My opinion is contingent upon the current situation being approximately maintained – i.e. that open source models trail the capabilities of the leading labs by a significant margin.
[Read More]