Publications

Here is a list of my current publications (* means equal contribution). I will try to keep this list up to date, however an always up to date list can be found at my Google Scholar.

Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters (2024)
Vasudev Shyam, Jonathan Pilault, Emily Shepperd, Quentin Anthony, Beren Millidge
paper | code

Zyda: A 1.3 T Dataset for Open Language Modeling (2024)
Yury Tokpanov*, Beren Millidge*, Paolo Glorioso, Jonathan Pilault, Adam Ibrahim, James Whittington, Quentin Anthony
paper | code

Toward Conversational Agents with Context and Time Sensitive Long-term Memory (2024)
Nicholas Alonso, Tomas Figliolia, Anthony Ndirango, Beren Millidge
paper

Zamba: A Compact SSM Hybrid Model (2024)
Paolo Glorioso*, Quentin Anthony*, Yury Tokpanov*, James Whittington, Jonathan Pilault, Adam Ibrahim, Beren Millidge*
paper | code

A Review of Neuroscience-Inspired Machine Learning (2024)
Alexander Ororbia, Ankur Mali, Adam Kohan, Beren Millidge, Tommaso Salvatori
paper

Natural Induction: Spontaneous adaptive organisation without natural selection (2024)
Christopher L Buckley, Tim Lewens, Michael Levin, Beren Millidge, Alec Tschantz, Richard A Watson
paper

BlackMamba: Mixture of Experts for State Space Models (2024)
Quentin Anthony*, Yury Tokpanov*, Paolo Glorioso*, Beren Millidge*
paper | code

Collective Behaviour from Surprise Minimization (2023)
Conor Heins, Beren Millidge, Lancelot Da Costa, Richard Mann, Karl Friston, Iain Couzin
paper

Predictive Coding Networks for Temporal Prediction (2023)
Beren Millidge, Mufeng Tang, Mahyar Osanlouy, Rafal Bogacz
paper

Exploring Action-Centric Representations through the Lens of Rate-Distortion Theory (2023)
Miguel De Llanza Varona, Christopher Buckley, Beren Millidge
paper

Causal Inference via Predictive Coding (2023)
Tommaso Salvatori, Luca Pinchetti, Amine M’Charrak, Beren Millidge, Thomas Lukasiewicz
paper

Associative Memories in the Feature Space (2023)
Tommaso Salvatori, Beren Millidge, Yuhang Song, Rafal Bogacz
paper

From the free energy principle to a confederation of Bayesian mechanics. Reply to comments on” How particular is the physics of the free energy principle?” (2023)
Miguel Aguilera, Beren Millidge, Alexander Tschantz, Christopher Buckley
paper

Generalized Predictive Coding: Bayesian Inference in Static and Dynamic models (2022)
Andre Ofner, Beren Millidge, Sebastian Stober
paper

Recurrent predictive coding models for associative memory employing covariance learning (2022)
Mufeng Tang, Tommaso Salvatori, Beren Millidge, Yuhang Song, Thomas Lukasiewicz, Rafal Bogacz
paper

Incremental Predictive Coding: A Parallel and Fully Automatic Learning Algorithm (2022)
Tommaso Salvatori, Yuhang Song, Beren Millidge, Zhenghua Xu, Lei Sha, Cornelius Emde, Rafal Bogacz, Thomas Lukasiewicz
paper

Predictive Coding Beyond Gaussian Distributions (2022)
Luca Pinchetti, Tommaso Salvatori, Yordan Yordanov, Beren Millidge, Yuhang Song, Thomas Lukasiewicz
paper

Capsule Networks as Generative Models (2022)
Alex B Kiefer*, Beren Millidge*, Alexander Tschantz*, Christopher Buckley
paper | Alex’s code, my code

Preventing Deterioration of Classification Accuracy in Predictive Coding Networks (2022)
Paul F Kinghorn, Beren Millidge, Christopher L Buckley
paper

A Theoretical Framework for Inference and Learning in Predictive Coding Networks (2022)
Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, Rafal Bogacz
paper | code

Successor Representation Active Inference (2022)
Beren Millidge, Christopher L Buckley
paper | code

A Theoretical Framework for Inference Learning (2022)
Nick Alonso, Beren Millidge, Jeff Krichmar, Emre Neftci
paper | code

Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning (2022)
Beren Millidge, Yuhang Song, Tommaso Salvatori, Thomas Lukasiewicz, Rafal Bogacz
paper | code

On Bayesian Mechanis: A physics of and by beliefs (2022)
Maxwell JD Ramstead, Dalton AR Sakthivadivel, Conor Heins, Magnus Koudahl, Beren Millidge, Lancelot Da Costa, Brennan Klein, Karl J Friston
paper

Inferring Neural Activity Before Plasticity: A Foundation for Learning Beyond Backpropagation (2022)
Yuhang Song, Beren Millidge, Tommaso Salvatori,Thomas Lukasiewicz, Zhengua Xu, Rafal Bogacz
paper | code

Reward Bases: Instantaneous Reward Revaluation with Temporal Difference Learning (2022)
Beren Millidge, Mark Walton, Rafal Bogacz
paper | code

Hybrid Predictive Coding: Inferring, Fast and Slow (2022)
Alexander Tschantz*, Beren Millidge*, Anil Seth, Christopher Buckley.
paper | code

Predictive Coding: Towards a Future of Deep Learning beyond Backpropagation? (2022)
Beren Millidge*, Tommaso Salvatori*, Yuhang Song, Rafal Bogacz, Thomas Lukasiewicz
paper

Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Models (2022)
Beren Millidge, Tommaso Salvatori, Yuhang Song, Thomas Lukasiewicz, Rafal Bogacz
paper | code

Learning on Arbitrary Graph Topologies via Predictive Coding (2022)
Tommaso Salvatori, Luca Pinchetti, Beren Millidge, Yuhang Song, Rafal Bogacz, Thomas Lukasiewicz
paper

pymdp: A Python library for active inference in discrete state spaces (2022)
Conor Heins, Beren Millidge, Daphne Demekas, Brennan Klein, Karl Friston, Iain Couzin, Alexander Tschantz
paper | code

Active Inference in Robotics and Artificial Agents: Survey and Challenges (2021)
Pablo Lanillos, Cristian Meo, Corrado Pezzato, Ajith Anil Meera, Mohamed Baioumy, Wataru Ohata, Alexander Tschantz, Beren Millidge, Martijn Wisse, Christopher L. Buckley, Jun Tani
paper

Habitual and Reflective Control in Hierarchical Predictive Coding (2021)
Paul F Kinghorn, Beren Millidge, Christopher L Buckley
paper

A Mathematical Walkthrough and Discussion of the Free Energy Principle (2021)
Beren Millidge, Anil Seth, Christopher Buckley
paper

Predictive Coding: A Theoretical and Experimental Review (2021)
Beren Millidge, Anil Seth, Christoper Buckley
paper

Applications of the Free Energy Principle to Machine Learning and Neuroscience (2021)
Beren Millidge
paper | code

Online Reinforcement Learning with Sparse Rewards through an Active Inference Capsule (2021)
Alejandro Daniel Noel, Charel van Hoof, Beren Millidge
paper | code

Towards a Mathematical Theory of Abstraction (2021)
Beren Millidge
paper

How Particular is the Physics of the Free Energy Principle (2021)
Miguel Aguilera, Beren Millidge, Alexander Tschantz, Christopher Buckley
paper

Understanding the Origins of Information-Seeking Exploration in Probabilistic Objectives for Control (2021)
Beren Millidge, Alexander Tschantz, Anil Seth, Christopher Buckley
paper | code

Neural Kalman Filtering (2021)
Beren Millidge, Alexander Tschantz, Anil Seth, Christopher Buckley
paper | code

Sophisticated Active Inference: Simulating Anticipatory Affective Dynamics of Imagining Future Events (2020)
Casper Hesp, Alexander Tschantz, Beren Millidge, Maxwell Ramstead, Karl Friston, Ryan Smith
paper
Published in IWAI IEEE workshop on Active Inference

Investigating the Scalability and Biological-Plausibility of the Activation Relaxation Algorithm (2020)
Beren Millidge, Alexander Tschantz, Anil Seth, Christopher L Buckley
paper | code
Published in NeurIPS 2020 workshop on Backpropagation in the Brain

Relaxing the Constraints on Predictive Coding Models (2020)
Beren Millidge, Alexander Tschantz, Anil Seth, Christopher L Buckley
paper | code

Activation Relaxation: A Local Dynamical Approximation to Backprop in the Brain (2020)
Beren Millidge, Alexander Tschantz, Anil Seth, Christopher L Buckley
paper | code

The Acquisition of Culturally Patterned Attention Styles under Active Inference (2020)
Axel Constant, Alexander Tschantz, Beren Millidge, Felipe Criado-Boado, Luis M Martinez, Johannes Müller, Andy Clark.
paper | code

Control as Hybrid Inference (2020)
Alexander Tschantz, Beren Millidge, Anil Seth, Christopher Buckley
paper
Published in ICML 2020 workshop.

On the Relationship between Control as Inference and Active Inference (2020)
Beren Millidge, Alexander Tschantz, Anil Seth, Christopher Buckley
paper
Published in IWAI IEEE workshop on Active Inference

Reinforcement Learning as Iterative and Amortised Inference (2020)
Beren Millidge*, Alexander Tschantz*, Christopher Buckley
paper

Predictive Coding Approximates Backprop Along Arbitrary Computation Graphs (2020)
Beren Millidge, Alexander Tschantz, Christopher Buckley
paper | code

Curious Inferences: Reply to Sun & Firestone on the Dark Room Problem (2020)
Anil Seth, Beren Millidge, Christopher Buckley, Alexander Tschantz
paper
Published in Trends in Cognitive Science

Whence the Expected Free Energy (2020)
Beren Millidge, Alexander Tschantz, Christopher Buckley
paper
Published in Neural Computation

Reinforcement Learning Through Active Inference (2020)
Alexander Tschantz*, Beren Millidge*, Anil Seth, Christopher Buckley
paper | code
Published in Bridging AI and Cognitive Science (ICLR 2020) workshop

Deep Active Inference as Variational Policy Gradients (2019)
Beren Millidge
Published in Journal of Mathematical Psychology
paper | code

Combining Active Inference and Hierarchical Predictive Coding: A Tutorial Introduction and Case-Study (2019)
Beren Millidge
paper | code

Implementing Predictive Processing and Active Inference: Preliminary Steps and Results (2019)
Beren Millidge
paper | code

Vocal imitation can create acoustic attractors to guide mothers to pups in a crowded colony of Mexican free-tailed bats: A case-study of computational modelling in behavioural biology (2019)
Richard Shillcock, Beren Millidge, Andrea Ravignani
paper | code

Fixational Eye Movements: Data Augmentation for the Brain? (2019)
Beren Millidge
paper | code

Exploring infant vocal imitation in Tadarida brasiliensis mexicana (2019)
Richard Shillcock, Beren Millidge, Andrea Ravignani (2019)
Published in Neurobiology of Speech and Language
paper

A Predictive Processing Account of Bottom-Up Visual Saliency Using Cross-Predicting Autoencoders (2018)
Beren Millidge, Richard Shillcock
paper | code