Probing the Developmental Landscape of Neural Networks

Abstract

Do neural networks learn gradually and continuously, or does their learning occur in sudden, discontinuous shifts? In this talk, I will argue that neural network development occurs through distinct developmental stages, drawing an analogy with developmental biology. The learning process of neural networks can be seen as navigating a developmental landscape, in which developmental milestones are geometric structures that guide networks from an initial “pluripotent” state to a final “differentiated” form. To make this case, I will present the theoretical background behind this research, highlighting connections to algebraic geometry, statistical physics, and singular learning theory. In addition, I will present empirical evidence from recent papers that discover interpretable developmental stages in real-world systems. The talk will conclude by exploring potential future applications of these insights, particularly in interpreting the internal workings of neural networks.